It hasn't happened to me yet. I would never refer to myself that way. But lazy and small-minded terms of ignorance and abuse exist for any and all socially excluded or otherised groups. They say far more about the person using them than the person receiving them.
If someone whose opinion I actually cared about said it, sure, I'd be upset (because yes, it does carry baggage, it clearly has derogatory connotations). I'd also probably stop caring so much about their opinion. If some random said it... meh, I don't know. My response would depend on where my own self-image and self-esteem were sitting at the time, perhaps. It's the same with being out and seeing myself in mirrors/windows. If I feel good about myself, I don't care what anyone else thinks. If I don't, then other people's (real or perceived) negative reactions amplify my own insecurities.