My advise, just use the bathroom you feel most comfortable in! Who cares? If others are uncomfortable with someone using the bathroom, that's their problem, not yours! Sorry, but I think people really need to get over it, everyone needs to use the bathroom, it shouldn't be a big issue!
If you feel more comfortable using the women's if you don't feel you pass well enough, then use the women's! If you would rather use the men's because you ARE a guy, then use the men's! Sad fact is, when you're still in that awkward 'in between' stage, you can have people whining no matter which bathroom you use!
I wish the 'family bathrooms' would catch on in more places, they really seem like the best idea in my opinion! You'd think they'd catch on especially for all the people with children! (even though I thought it was so cool to be 'allowed' to go into the men's bathroom when I was out with my dad when I was little! ha)