I think part of it is that many people in the mainstream are only peripherally aware of TG/TS, mainly in the context of fetish porn or the occasional unfortunate XY transwoman who is actually a wonderful person but who simply doesn't pass and is therefore unfairly categorized by the observer as a "linebacker in a dress"...and therefore somehow abnormal. They don't have any idea that they may have interacted with any number of transpeople in their day-to-day lives and simply didn't notice because the transperson passed convincingly and "acted normal".
Since it is the boundary exceptions that are usually seen by the public, the public tends to stereotype all of us as fitting those boundary cases. We humans seem to love stereotypes, whether sexual, gender, racial, functional, or even political.
I also think there tends to be some bias between transmen and transwomen in the public eye as well, just because transmen are a bit more...invisible, socially speaking. A transman who passes poorly may be perceived as younger, gay, or effeminate, but still male. Even a chest too large to bind reasonably can be made passable with a relatively simple surgery, if money permits. A transwoman who passes poorly, however, tends to be more obvious due to bone structure and deeper natural vocal resonance. (There will always be exceptions to every rule, of course.)
The best thing we can do to counter this is to continue working on promoting awareness and acceptance until the day when the average "Joe Blow" on the street understands that it isn't about sex; it's about quality of life. It's about identity. In many cases, it's about our own sanity.
