Mostly it's because you probably didn't have a sexual attraction to him that overrode any innately biased feelings, but those feelings probably come from a mixture of self-doubt and sociological expectations. In other words, you probably subconsciously hated that this man was being freely feminine on the inside with the type of outer shell you may have been previously self-conscious about. Either that, or you were raised by a family or surrounding that repeatedly told you it was wrong for a masculine looking person to express any type of female or homosexual side. There's absolutely no such thing as a person behaving wrongly unless they do something against the law; any type of negative feelings toward someone else not based on law comes from a psychological place based in past experience or the sense of self.
It's why I hate looking identical to the actor to played Conan on the recent film; people just get plain grossed out and hateful when I act like the woman I know I am, or even if I act a bit gay. People usually cannot separate an appearance from a personality, and when the two contradict, it usually creates anger, hate, disgust, discrimination, laughs, jokes, and all that. To stop this type of discrimination and anger, I underwent two nasal surgeries where the doctors mutilated the structures of my nose against my will, causing it to collapse. And now, I plan on spending $18,000 to correct that feature as well as undergoing brow bone reduction and other treatments. All to make people see me for the woman I am, all to make people automatically accept my personality, not laugh or turn angry at it.
Not to say I'm perfect where discrimination is concerned; I personally think many gay people are plain annoying too. Usually though, it's only because I hate hyper people, because my parents used to yell at me constantly if I ever did anything hyper or upbeat as a child, so I'm always a bit jealous of them. The only time I can ever stand hyper annoying people is if they're sexy in my eyes, then I'll somehow magically put up with them all day long. Like some years back, there was this annoying gay kid in my high school, he had the most annoying voice and camp attitude ever; it was annoying because everyone loved him and not me, just because he was much hotter than me. Ends up I had sex with him like a year later, forgot all about how annoying he was while it was going on!