Power of human eyes and brain devoted to vision are staggering. We are a very much vision bound species. Our ability to decipher facial details is astonishing and goes deeper than skin deep. In fact we are able by default to subconsciously read a state of physical/mental health and genetic predisposition. Our evolution is to thank for such abilities much in our Transgender disappointment. Jesse, you are absolutely correct in assessment you have concluded. Female is read female because of the package of all necessary characteristics fitting in as a whole. Ability to decipher facial character makes for the face being most important in passability, however if the rest of the package does not conform to the first observation, observer will question immediately validity of the first assessment, picking apart all aspects. "Hence a double stair"! Observer looks onto a good looking TG, brain assesses input, makes assumption of interest, if the look is passable but not interesting observer goes on, if the interest arises the second deeper assessment will follow.
Mannerism is important but if mannerism has no backing in facial and body geometry it will not make you pass.
I am gravely disappointed for my self starting transition so late and have come to the conclusion that full passability is mostly reserved for those young ones brave and determined. Some of us older ones posses "I don't care what others say" attitude which feels like passability is in the attitude, but it is just wishful thinking on their part. I believe, at least to me, passability involves fooling someone elses powerful visual brain to read you as female when you are not( genetically that is). If passability was ego centered, meaning, we needed to fool our selves in believing we are females, we wouldn't have this discussion in the first place. It all comes down to the most basic and eons old human need, To belong! To be accepted as you are, in our case we have to swim against the current of human basic knowledge and validate to the world that we are as normal as anyone out there, guess what?