I have this problem, I pass and I get weird looks in the bathroom as well as having people check the sign and once a pub landlord try and drag me out -- fun times! -- but I'm not out as male. I try and judge times when no one else is using the toilets, or scout out places that have gender neutral bathrooms. I don't like using the wheelchair accessible ones because my worry of having someone get upset at me for using those as an able-bodied person is greater than getting looked at weird in the women's, but to someone more sensible than me they'd probably work just as well. If it comes to the worst I get in and get out, making sure I've got my driving license in my hand so I can "prove" that I'm "Miss XXX".
I've just thought, could you say to them, "Nah I'm okay, I'll just wait out here for you," when they want to use the toilet, and then when they go inside dive into the mens? You should be out before them unless there's a queue.