While I can only speak for myself, my best guess is a great deal of social conditioning.
My background is a rather violent one, though. Every time I try to say anything out loud, there's a five-year-old in me that doesn't want to get hurt anymore. My family tried to "fix" me, and they did fix me at least talking about it. To this day, I still barely can, even to the 20+ therapists I've been to over the years.
While I dearly hope I'm an extreme case in that regard, even seeing that kind of treatment of other people like us can go a long way to creating that fear. Absolutely nobody wants that treatment for themselves, and we've all seen or heard of at least a few instances where people were treated very, very badly just for being trans. Many didn't survive.
So, even if it's a false dilemma, that's the best answer I can give to this question. If we're already living in relative peace and safety, that fear of the unknown can be powerful. Because the unknown could be very, very bad.
Better the devil you know than the devil you don't.