Honestly, I think a lot of our paranoia comes from only a few things:
1. Not so smart decisions made by other trans people. Such as assuming that there will be no problems with sex with someone. You know, it'll be ok. He likes me, he thinks I'm hot. He won't mind me having the wrong parts. Prime example: Gwen Araujo. She gambled on having sex without disclosing her trans status, and unfortunately it got her killed.
2. We think people care. Really, I think like 80% of the public is more concerned about what is happening in their own life, and too busy with their own thoughts to really give more than a passing glance.
3. Of course low confidence. Thinking we aren't presenting well enough is almost as bad as forgetting to put on makeup or maybe a wig. I must say I pride myself in not having this problem, but I think a lot of trans people do.
just a thought.