Likely 'safer' in the men's room. Why? well the average male would rather die than have someone think he was afraid of your presence there.
Whereas in the ladies room, well of course the media does a nice job of trying to portray us MTF types as something of a certain danger eh. I mean, I suppose women are just so defenseless I guess in the eyes of the media.
And considering how homophobic too much of male society is, the thing is it is so much a rule of never using the urinal next to a guy if you can use one a spot removed from the next dude. And they are so busy NOT looking, that odds are they will have trouble knowing if you are or are not. So best advice is to just do nothing to invite attention, and they likely will be doing their best to not look like a gay person on the hunt.
Men for instance do their best to say as little as possible in the rest room. So say nothing, look nowhere and be in and out in as short a time, and you will likely seem like one of them in behaviour at least.