I agree with Cindy: just use the men's. If you pass well enough to be given grief in the ladies', you're man enough to use the men's.

I'm not 'man' enough to use the men's either, but I've been doing it anyway since last year. Confidence is key. As Cindy says, men's rooms are very different from the ladies': in the ladies' rooms, women tend to check each other out, make eye contact with each other, hold conversations, etc. which is why that 20-something clocked you and gave you a hard time.
Men don't tend to look at each other at all. In fact, they almost seem allergic to each other in the rest room. Generally speaking, men go into the men's room to do their business without looking at anyone else there or talking to them, and then they get the heck out as quickly as possible. Very often, they don't even stay long enough to wash their hands before leaving.
So just do what I do. March confidently into the men's room, go directly to one of the stalls, and do your business there. If you're worried about the sound you make when you pee, just put some toilet paper in the bowl before you start. If you're worried about people noticing you've gone in the stall just to pee instead of using a urinal, just wait in the cubicle for an extra minute so it seems you've had a number 2, and if the toilet has a dual flush (a small one for a number one, and a big one for a number 2) always use the big one. If the stalls are full, just wash your hands and leave (as if you'd only gone in to wash your hands in the first place) and try again later.
A good place to start getting used to the men's is Starbucks: they tend to have single-stall toilets so you'll be on your own in the men's room. Then work your way up to the multi-stall toilets they tend to have in shopping malls etc. when you feel more confident.
I've never been confronted in the men's toilets (yet), but just in case, I keep a copy of a letter from my Gender Therapist that identifies me as a female-to-male transsexual who must be treated as male for all legal purposes. It gives me the extra confidence I need.
If some guy ever asks you whether you're in the right room, just say 'sure!' and carry on doing what you're doing. You don't have to engage anyone else in conversation. They have no more right to know what you're doing in a toilet than you have to know what
they're doing there.