This is has got to be the one shared topic that is almost the same for M->F and F->M.
I always try to find the unisex bathroom, and I keep a list of them for the places I go to or have been at....even if I'm out on the town, I always watch for them, for future reference.
Although it seems that the bathroom passing issue tends to be mostly in our head, it only takes one bad incident or a few "weird" looks or comments, before you are totally second guessing which one to use.
The one thing that you have going for you Aiden is that men generally try to get in, get out, and not talk/look around too much when they are in a public restroom; so at least you have that going for you. Men usually don't hang out, wait for their buddies, or talk in the restroom; however, if you find that they are, this is when they will likely make a comment or worse. Conseqently, I would use that as a warning sign to keep my guard up.
Personally, in a two day span a couple of months ago, I had comments made to me in the mens room and the womens room(and I was presenting in the appropriate gender mode for each of the restrooms I was in), so it can be really hard at times. Unfortunately, that experience has driven me to not be comfortable in either restroom. However, I'm sure like anything in life, that practice will make it easier, and less "scary".