My theory on that, which is very subjective with no rigorous study conducted, really anecdotal. Is that some people just don't really look into the details a lot, and accept you at face value(just accept you), while a tiny majority, are able to decipher, if one is trans or not, due to a variety of reason, or it could be projection on your behalf.
Some of the other reasons is that:
1. Deep down, they are just curious about trans people, but don't really articulate it to anybody, maybe love and/or hate feeling.
2. Some clues, which maybe missed by some, but not by others are, facial structure, body structure, feet, hands, voice, mannerism, et al. physical cues and maybe societal and cultural cues.
3. Somebody else may know the person is trans and investigate that person.
4. Projection
JMAT(Just My Anecdotal Theory)