It sucks when that happens. I've had that happen (well, not asking people but the whole getting clocked multiple times in a day thing).
Try not to let it get you down, but also try not to hold it against them. You'll just end up a bitter man that way.
When it happens to me, I have to step back and remember that I was born into an XX shell after all. They're just reacting to the shell. It's nothing personal. They don't know you, and they don't know your actual gender identity.
You didn't "fail". Different people see different things. As long as you're being yourself, then you're doing just fine. You're just doing the best with what you've got, and they're doing the best with the social protocols they know of. Nobody is "at fault" or in the wrong, it's just a little misunderstanding not worth correcting (unless you plan on making them a part of your life).