I feel you here, I took quite a long time to pass and my confidence was really screwed up by it. Even with the bathroom thing.
Honestly, if you're passing well enough in day to day things, it's a good time to try using the men's. I was scared to death for months, dashing into the stall and hoping to god no one would pass me as I left. I still feel scared sometimes...the "what ifs" but at some point, you will have to take the plunge. I am at the point where I think I'd get kicked out of the women's and I've used the men's exclusively for 5 months or so. I understand not feeling comfortable enough. If you really aren't passing consistently and the women's feels safer, stay there. If you are passing often, consider trying the men's. People can say "you're a man, use the men's" but if you don't feel the world sees male, you shouldn't take risks you are uncomfortable with.