I've only ever been with one girl, and her I married... and later divorced. As annajasmine commented, I also felt distinctly uncomfortable, and since splitting up, I've been celibate - going on 6 years now. I'd never had much of a sex-drive, but since then I'm for all intents and purposes a-sexual. I find both sexes mildly attractive, but only after I get to know the person and start sizing him/her up as a soul-mate. I'm much more interested in whether there's a ... connection there, and frankly sex is the last thing on my mind.
I suppose it's a combination of being very uncomfortable with the body you are in plus general loneliness and a need for companionship, but who knows - maybe it's just the lack of testosterone.