Ok so I have yet to enter a woman's restroom

i am still fearful of doing so. I have technically been to one at our church when it was not busy and also used the unisex there at church. I have been sticking with unisex restrooms.
Same goes for the YMCA I am a member at here in San Francisco. I use the family rooms which also double as a unisex room and special needs room. I had a lengthy talk before hand with the director of that facility. He totally respects me and understood so made it clear to staff I could also use the family room on the woman's side (which I have but it has no water pressure....go figure).
So here is my dilmea though. In general am I required by law to use a public facility or a private one I pay for on the mens side? And if its legal for me to do so, when should I try it? Because I feel like I am ready but I wanted to know the entire skinny when it came to restrooms, pools etc etc. Especially bathrooms where there are multiple people is an issue. I am getting uncomfortable using certain mens ones that have large crowds like the library.