You'd be surprised how many of those opinions when not being passable, judge someone as androgynous (ie, they don't know), and in many cases, that's why they stare. Gendering people is something people do automatically, at-a-glance, not something people have to think about consciously most of the time.
Now someone who's androgynous throws a wrench in those gears, they don't know, they're not sure. It could be a lack of clues either way, or heavily contradicting ones, clothes notwithstanding. Clothes can lead to confusion sometimes, but they're a smaller factor in the balance, because you can change clothes any time, and people know that (even if it's harder to visualize).
Hence why some pass easily in t-shirts and pants, while others can't pass in high heels and a skirt. Clothes are secondary in passing, they're the cherry on top, and its optional (not optional not to wear any, but to wear very-gendered clothes, like a dress or a skirt or a cute top is optional).
My point is, what people think when they stare is not necessarily "Ah, it's a man!", but "Is it a man or a woman?" most of the time, morbid curiosity (like an accident on the roadside draws) because it goes out of their experience, or they notice it for the first time.
Some decide either way, male or female, and some remain indecisive after watching longer. I caused indecision in people when I presented as a guy.
Some cues pointed to mostly female:
small frame
very long hair
soft features (soft skin, no roughness)
lack of body hair
lack of muscle
lack of a stuble (if I shave every day I look normal)
Some cues pointed to mostly male:
non-high-pitched voice, especially coupled with such a youthful-looking body (androgynous voice, who could pass as more mature and still female, or as male but young)
masculine clothes (very baggy t-shirts and pants, always in sneakers 'of boyish colors')
lack of much attention to hair-grooming
and that's at-a-glance, they didn't have a long conversation with me, they might have seen me walking down the street, or overheard me order something at a restaurant they're sitting at, or something
That was pre-transition.
Since starting full-time there was even more confusion, more staring...but over time it faded, after six months I fit in as just any girl, physically, no one could pick me out of a crowd, no one would stare. I 'lost' my physical androgeny.
Remember though, that confidence is one of the better things of passing. If you're confident, you'll pass better. Being scared or nervous/anxious draws attention to you and more scrutiny is applied. Such things easily overlooked otherwisely.
