This is latent transphobia. I really wouldn't take it personally. These people are just uneducateable! Kudos to you, though for holding your own!
It would annoy me too, to be honest, but don't let them win. They are seriously misinformed. Bigotry is not worthy of your being upset. Chin up