Interesting Frances, I was just talking about this very thing with a trans friend a couple days ago.
Here is where our valley is:
* People can deal with cis folks. If they read us as cis, no problem.
* People can deal with trans folks. If they clock us or read us as clearly trans, no problem.
* However if we're almost passable, but there's just the slightest thing off - something that maybe people notice subconsciously but not consciously - it makes them uncomfortable. They stare and become uncomfortable around you because they're just not sure what's wrong.
Before I got my current wig, and before HRT had done all its work, I would get a lot of stares. I noticed my clearly non-passable friends never got similar stares. When I started passing better, the stares went away.
Great topic.