From what you're saying, I'd say you're on to something with intersex.
A few years ago I did some heavy research into being intersex when some things came to light surrounding my mom's health while she was pregnant with me and some curious info that my aunt sent me (she sent me a birth announcement card where it was all "it's a boy" blue color but had my baby picture on it and also a letter where my mom said she had an ultrasound and was having a boy). I am one of those people who came out of the shoot with the larger than average external junk (for a female), I've had "reproductive" problems throughout my life but never had what could be considered "normal" female puberty or monthly crap. And of course, since day 1 I've just thought I was male (very upsetting learning the truth about the physical body when I was like, 3 or so – but nonetheless, that's the way my brain is set up). Thank the universe I don't have a genetic disease to deal with it on top of it all! But I truly believe that there's a ton of different variations to intersex – some are very much physical things that can be seen or tested, but I personally believe that being transsexual very much does have a biological brain component (not just a mental component). When I was doing my own research I ran in to a number of articles on what hormones do to developing fetuses. My mom's ovaries had stopped functioning few years before I was born. It was only because of another operation she had and the doctors operated on her ovaries in an attempt to make them function. I believe she had PCOS that went undiagnosed. I was conceived not long after her operation and my personal thought on it was that genetically I may be an XX (never had a genetic text but that's my assumption) but while my body started to grow out of those two cells I was bathed in the T that was coursing through my mom's system. Hormones are powerful stuff and they don't always conform, for lack of a more medical term, to genetics. So even though I'm not XY and didn't get all those proper "signals" in the womb, I feel that the excess hormone had a similar influence and it changed me from being a normal, functioning female.
My belief is that being transsexual can happen in the womb, just like a myriad of intersex conditions, genetic diseases ... plenty of not "normal" or "average" things that can happen while a baby is being formed. I hope science really does validate this some day.