I have had several scary experiences using the "men's room" while attempting to present as "male" and while I get sir'd all the time, other restroom users had very different thoughts. What is required, 2 pieces of clothing? Hair? Nail polish? Voice? I had never felt the safety issue before being on HRT for another few months and things began to be unhidden.
In the end, I would actually feel way more comfortable with a gender neutral restroom to use and I seek those out or hold it. There is not going to be any sort of resolution to this as we are fighting the binary only crowd and have no options other than the options they allow us to have in bathroom choice.