Feelings of inadequacy are the worst part of it for me...
That feeling that no matter how much I've been on hormones, they won't ever change the fact that I went through a male puberty. And thus I'll always feel like less of a woman because I'll always have wider shoulders, a changed voice, a receded hairline, bigger pores and less skin softness, bigger hands, bigger feet, and a stockier build with big bones.
And this is from someone who does pass. I still feel bad about my appearance a lot, and still feel like I'll still never be the woman that I really wish I could be. It's that pain of what could have been that's the worst part.
Getting fired from jobs twice as a result of being trans, and being rejected from a job interview because of my nervousness as a result of that too, has done my mental state no favors either. In fact, it's probably a cause. It's hard to have self-esteem about being trans when it came damned close to destroying my entire working life. That's made me so damned afraid of other people's judgment, and so afraid of messing up.