i understand how much it may hurt others to be asked about their gender in this way, even if i'm a type of person who'd love to be able to confuse people enough to make them ask (not that i'd give any clear answer tho..)
my experience with this way of asking a question is that the person asking knows how offensive it may be, and that this is exactly the type of curiosity which is known to kill cats.
i've personally been asked like this only once, though about my ethnicity (i'm sami, could be likened to native americans, only difference is that you often can't distinguish between sami and norwegians based on looks). this was in one of the most prejudistic places i know about, only people that are lower than the sami are homosexuals, transgendered, and the like. asking about it and getting it wrong (or if the person is a closet case) could very well get you near killed. so it can be just as scary to ask, as it is hurtful to be asked
and now i already forgot what i wanted to say with this... well, just consider the possibility whoever's asking really isn't trying to hurt you, but only is a very curious label addict cat