I've always looked pretty male, even though I'm pre-T. I've lived in a big, tolerant city, a small village where everyone knew each other, and a far less tolerant town 25 miles north of the big, tolerant city where there were some distinctly dodgy areas.
I never had a problem in any of them apart from a few kids shouting stupid things at me and my then-girlfriend when we went to pick up a curry one night. But I think they would have shouted at anyone who was non-white, muslim, disabled, etc. They were just yobs.
I think a lot of it is about how confident you are, how you carry yourself, and how much crap you're prepared, or not prepared, to take.