That is crazy! I live in Dallas, Texas and no one has ever even commented anything negative about my gender or sexuality. Of course it is possible that I radiate a sense of "I will be happy to punch you if you mess with me"--I am a pretty alpha person who is more than a little aggressive!
Honestly I think you pass really well. No, you don't look like some super hulky bro or whatever, but you look like a guy. Maybe you are getting hassled because they think you're too slim and untraditionally masculine as a man and don't like that? (I am not sure what word they called you....)