I've never had problems with restrooms, though before I was passable, I tended to get in and out as quickly as I could without anyone getting a good look at me. There certainly are enough horror stories posted in these forums to instill a healthy respect for the issues that can arise in one.
Now I have an attitude. Yes, I mostly pass, but I'm trans and I'm proud and I DESERVE to use spaces appropriate to my gender. Anyone who would interfere with that, will very quickly be put on the defensive.