As a girl who wants to someday crossdress as a male, but still use the female bathrooms while dressed up, this is a subject I'm also worried about.
My plan is to act as if I belong, even if I don't look like it. Be confident and relaxed, as if you're not worried about being seen as someone in the wrong restroom. Even if you are worried. If someone saids "This is the men's/women's room", give a short calm response that states you know what bathroom you're in. Just go in and out, don't rush but don't spend anymore time then you need to.
People can and will pick up on a person's discomfort of being in a place they don't belong, especially when it's questionable if the person belongs or not. The person's discomfort reinforces the idea the person is in the wrong place, and also makes him or her seem like easy prey. By appearing confident and relaxed in your environment you will look more like you do belong, and it will be less likely for someone to try and intimidate you. Hence the importance of your response to "You know this is the woman's restroom?" By asking you that question the asker is trying to create an opening. You want to nip the bud before it grows. Make sure your answer gives the impression that you know very well what you are doing. Be polite when answering the question, look the person in the eye. keep the answer short, and after you answer go back to what you were doing. Even if what you were doing was staring at a wall. This gives the impression that you don't feel like talking and leaves little room for the conversation to continue. If they ask again, simply repeat your first answer more firmly. I think at this point only the persistent or the aggressive would continue.
Don't spend anymore time in the restroom then you need to. The less time you spend the less access others have to question you or start harassing you and the less time you have to endure they're stares. Don't rush though, since it will make you seem like you are uncomfortable and tense. Once you are out of the restroom don't get hung up too much on what happened in there. All you did was your business, like everyone else does in the bathroom.
When something isn't where it belongs, it's human nature to wonder WHY it's there. So women automatically try and figure out what a man's intensions are for being in the women's restroom. The possibility the man is a pervert will cross their mind. Plus the man in their eyes is a rule breaker aka unpredictable, they can't tell how far the man will go. It's similar to the way you'd react if there was a random bear in the restroom, how likely is it that you are not going to keep your eye on it. I don't know if the reaction to a women in men's restroom is the similar, doesn't seem that way. Women seem to have a much easier time using the mens restroom. Maybe because women don't seem to be much of a threat. A good time to do so is when there's a long line in front of the girl's bathroom. I've seen women in dresses casually use the mens restroom because they don't want to wait. Another reason you can give for being a women in the mens restroom, is that the women restroom was really messy/gross.
You might want try having a buddy with you in the restroom. If there is already someone that is accepting of a man in the restroom, the others may follow suit.
These are all theories and ideas, since I have yet to have a chance to pass as a male. Does it seem right?