I'm afraid I'm going to use a cliche here, so please bear with me.
Haters gonna hate.
But seriously, many of the reasons pointed out by everyone are more than likely true. We are "outside of the cis male/cis female box," thus something that people are sometimes afraid of. We are people that realize that we are the opposite of what our genitalia are, what our ambiguous genitalia are, or someone that wants to be somewhere in the middle of what our genitalia is.
We are different, and until they either research "transgender," talk to someone that's transgender or realizes that they are transgender, some people choose to fear us.
Hopefully, those people will realize that we are just people. That we just want to live out our lives, nothing more.
Ryuichi