Hi guys, I'm MTF, but I grew up as a man. So here's my take:
Mens bathrooms are very different from ladies bathrooms. Men don't typically talk to each other, and there's virtually no chance of having to stand in a line. Men get in and out, and many don't even stop to wash their hands. So you should be fine, and will probably never even be noticed, especially since you say that you pass most of the time.
Just one girl's opinion 🙂