Absolutely.
I think for the most part people WANT to mean well, and feel bad when they've made a hurtful mistake. (I want to believe that anyways.) Part of my brain always returns to the thought that as soon as I'm out of earshot or view, if they've correctly gendered me or not, they're going to run off to giggle about me to a coworker or friend 'DID YOU SEE THAT FREAK?!'
As strong as I try to be about it, a single misgendering DOES throw my whole day, even up to a week after. I'll literally dwell on what the hell it was about me that gave it all away when in my eyes there's been nothing. So I'll start CREATING things, maybe the clothes I was wearing, maybe it's my bone structure, maybe my voice wasn't spot on that day. It's uncomfortable, and awkward, and often, it's 100% my own doubt.
All of that is on me. I think we do the most damage to our own self esteem. It takes very little prompting from others.