I looked somewhat like a boy for most of my life, and I had a deep voice, and I was lots and lots of times confronted in women's restrooms. Most of the time people thought I had made a mistake. They would sheepishly inform me that I was in the wrong bathroom. Sometimes I just finished my business and let them think they were right, and sometimes I corrected them or argued with them. I do think we have culturally made a mountain out of a molehill with the perceived threat of men using women's facilities. A lot more suffering is caused by this idea about danger than is caused by the actual danger itself.