Exactly, Hugh. All of the groundwork gets laid in utero, in particular during the 8th-16th week of pregnancy when the sex glands begin their descent from the base of the brain down to the pelvis. It is at this time that sexual hormone production begins in earnest and anything that disrupts that production or alters the "normal" ratio of testosterone to estrogen (females produce testosterone too, just the ratios are very different) can cause variations, particularly in the brain itself because the brain is more isolated from the rest of the body due to the brain-blood barrier.
For example, at the age of 38 I was diagnosed with a media-stinal germ cell tumor. This tumor appears to be the direct result of sex gland material that gets deposited in the chest (media-stinal) during the descent of the sex glands during the 8th-16th weeks of pregnancy. My oncologist, Dr. Robert Amato, one of the leading researchers in the world at that time on media-stinal germ cell tumors, said that in his data every single patient had mothers who were heavy smokers during pregnancy. This latent germ cell material then lays dormant in the chest until it randomly gets activated and becomes one of the fastest growing tumors in the cancer world.
When I began therapy with my therapist years later, I reached out to Dr. Amato, who was no longer at MD Anderson Cancer Center but over a Baylor and asked him if the mechanism that would cause tissue to break off and form a latent germ cell tumor would also cause disruptions in production of testosterone. His reply was "Absolutely!" And that it might take weeks to return to normal levels of hormonal production.
Though I cannot absolutely prove it, the data now strongly suggests that my own transgender nature, which I've had since very young is a direct result of this.
Also, there are XX males, who have what is known as de la Chapelle syndrome. It is caused by unequal crossing over between X and Y chromosomes during meiosis in the father, which results in the X chromosome containing the normally-male SRY gene. When this X combines with a normal X from the mother during fertilization, the result is an XX male.
This syndrome occurs in approximately four or five in 100,000 individuals, making it less common than Klinefelter syndrome.
According to research at the University of Oklahoma health science centers, most XX males are not stereotypically feminine and are typical boys and men although other reports suggest that facial hair growth is usually poor and libido is diminished, with notable exceptions.
So once again, genes themselves are not the determinants but only the blueprints and if those blueprints are implemented in a non-standard way, variations result.
Since we know that extremes of hormonal variations, such as de la Chapelle babies and AIS babies are possible, does it make sense to believe that less extreme variations might occur that explain being trans? Yes, that makes a lot of sense.
And we discover that DES babies (from the 1940s and 1950s when mothers were routinely administered DES during pregnancy which has since been banned precisely because it interferes with hormonal actions) have extremely high rates of being trans, being gay, and being intersex.
So all of the evidence begins to point towards hormonal variations as a root condition in being trans. And then the neurobiological research that I provided above shows that researchers now accept and began looking for proof of this in the brain and have found it in abundance.
Note that this does not discount the impact of "nurture" in this equation, but instead simply asserts that in the majority of cases a hormonal variation during pregnancy is likely what created the preconditions to being trans later in life. It would not surprise me at all to discover that this precondition is often coupled to other "nurture" factors but I would also wager money that in the majority of trans cases, the underlying brain structures are exactly as expected - for MtFs a female structured brain in a male body, and for FtMs a male structured brain in a female body.
