I had that problem, and here's why:
1. The lack of social support by society in general causes huge amounts of stress. Not surprising, as it is as scary facing the thought of constant social rejection for the rest of your life.
2. MOST importantly, this isn't very real or as real as it should be, because the surgical and medical techniques which are currently being used are so primitive. Part of the reason it's hard to gain acceptance in the first place is because we can't fit in, we can't fit in because the medical procedures are lacking, and the medical procedures are lacking because:
A. Society doesn't care enough to help us because we're heavily stigmatized.
B. Transgender people tend to be poor, have less access to education, small in number, and are indoctrinated into believing that better medical care (if all transgender people could pass/be cis) would eliminate diversity and acceptance. So the little influence that we do have is essentially discarded.
It causes tons of stress because knowing all of that I feel like the choice of transitioning is a lose/lose situation. This postpones my transition and creates dysphoria.
I hope this helps, if only to understand yourself!