I think it's mostly related to life experiences. In my childhood, my female cousins, and some of my very first classmates, excluded me, mainly because I was a boy trying to fit in with girls, and also because I have the habit of just doing everything wrong when it comes to social contacts... They rejected me and made me feel like I was "out of their clique". To be fair, though, most of my cousins are mean to begin with.
So I stuck with boys, mostly. Girls wanted nothing of me, anyway, and my association with them was always discouraged by my father. And well, I always need to get used to things to try them. Even now, almost always, someone has to take my hand and do something with me, explaining each step, at least once, or I won't feel comfortable doing it alone.
Life hadn't taught me how to be friends with girls. Indeed, most summers, my mother was subscribing me to a summer day camp that had the weird idea of separating boys and girls at almost 100 %. Since such a place is the best occasion to socialise...
So now, even though I don't really feel like I fit with them, I'm mostly okay with boys. Boys, that is. Men, they've always repugnated me (though of course that's a strong word to describe how I feel now - even I evolve!), because they were what I absolutely did not want to become. And since I have this habit of always needing to put an absolute value on things because at some point I stopped allowing myself to have an opinion, men became evil in my mind.
So with my male classmates who still somewhat look like boys/teenagers, I'm pretty comfortable. But those who look more like men - one of them, for example, is fat, hairy and bearded - I don't like to be with them much, out of my will. I can get used to them, and for some of them, I do, but even in those cases, I feel the need to make efforts to interact with them.
Almost the same rule applies to men. The softer, the nicer they seem, the more not-so-masculine or young their voice sound, the higher their "baby face factor", the less hairy or bearded they are, the less uncomfortable or fearful I am.
And similarly, the more females still look like girls/teenagers, the more I'm uncomfortable with them, because they and I once teamed up to teach my brain not to be with them. At high school, it was roughly the same. Most of my female friends were, in appearance and mind, more like women, whilst the few girls I stayed away from were younger-looking.
The problem is much less present now since females grow to look very much like their adult selves younger, and it was already not so present in high school for the same reasons, but still: I'm always more comfortable, in terms of first impression, with girls who appear older and/or more mature than I.
And of course, I've always felt much more comfortable with women, so much that considering my average not-so-social self, it's surprising.
So uhm, yes. My life has shaped me to be with boys and women, and away from girls and men, and that's how I still am to a degree.
I think being a transsexual and thus loathing to become a man has a relatively high chance of triggering a fear of men, but it greatly depends on how you lived it. In the end, all of that is just a persistence of who we learned to fear or like in the past.
