I used to ask to people if their parents didn't teach them the difference...
I actually got asked that by a whole bunch of kids the other day on a job - which I'd like to think at almost 11months on T was not still an issue but I'll forgive them, they're kids.. I just said boy and kept doing my job - they seemed happy enough with that.
Chances are if you're passing enough to make people actually ask that question you can probably get away with saying boy. And whats the worst that can happen anyways?? I'd like to think that they are not going to strip you down in public to 'prove' their point...
And as you say - you can always walk away.
Though I do like the yes or no answer

lol