I get the same reaction from my ex periodically.
To me it shows how much stock we all place in appearances. The person I am inside, my experiences, loves and peeves haven't changed. How I present myself to the world has, in a big way, causing those around me to perceive the male me as being "gone".
It profoundly bothers me when someone says something like that. It means they're seeing me on a superficial level and not the person I am inside.