Discrimination can be annoying or devastating. It all depends who does it. If you're trying to get a seat at a local restaurant and the owner denies you, then it's annoying. If you're looking for a job or a place to live, then it can be devastating.
Society is still very bi-gendered. Emphasize "VERY". Most people's minds go to wild, absurd images when they can't immediately determine gender. Even more so when they know you've crossed genders. Remember, the average IQ is something like 93. That means about half the people fall below that mark. If they aren't prone to learning they are probably prone to irrational behavior when confronted with something foreign to them. (Rule of 93) The irrational is magnified with some of the Jerry Springer fan-types and irrational behavior cannot be accurately predicted. Shoot first, ask questions later.
Now, if you find yourself pinned up against the wall in the mens bathroom, odds are you broke a male rule of engagement. Unless you outed yourself or simply cannot pass, you probably made some sort of contact with the guy, even just a sidelong glance, and he's bigger than you. BIG NO-NO! In the mens bathroom you walk in, do your business and walk out. You may be able to get away with light conversation while washing your hands but for the most part you should NEVER make contact. It's a very difference experience from the womens bathroom. Think battlefront when entering a mens bathroom. And BTW, the Rule of 93 applies here too.