My school has quite a few "man/woman/handicapped" single bathrooms, which are practical, and in smaller my building, strictly because there's nothing else but still, the only bathroom is unisex.
For real, there is strictly no point in gender-specific bathrooms. >.> The only thing that makes new students uneasy is that they've never seen one and they've been stupidly convinced, like the rest of the population, that there is a good reason why there are seperate bathrooms.
Anyway, uhm, for your question... don't worry about the women's bathroom. When I looked pretty androgynous, I did a few tests and noticed that I got weird looks and even "wrong bathroom, lady" (from old men; old men have a very very narrow idea of what a man can look like) in the men's room, but nothing in the women's bathroom. A little weird look once in a while at most.