I can pass only some of the time (pre-T) so I use the ladies' the majority of the time. If I'm actively trying to pass as male, and if I know the restroom isn't usually crowded, then I'll use the men's.
Funny enough, I've gotten funny looks from patrons in both facilities. Once in an airport, I wasn't even trying to pass-- no chest binding, voice altering, nothing-- and I was initially redirected to the men's by a worker until I indicated my tumors. I would have taken her directions, only I was with my Mum and it would've been really awkward had she seen me emerge from the men's, haha.