Being our own harshest critics, let me give you the following story as an example of how we make things out in the worst possible way...
I was with another transwoman and a few natal females as well. We were out in a public space (restaurant) when some men began really looking at my trans acquaintance. My interpretation of what I was seeing was that they were looking at her sexually, as a visually attractive woman. She, however, felt she had been clocked and was embarrassed by it and sure that was the explanation. But then both natal females chimed in as well, assuring her that they too were very sure the men had basically been ogling her sexually. They hadn't clocked her at all but instead found her attractive.
This even further confused her because she was sure she never "passed" yet there she was all evening passing just fine and getting cisgender male attention as well.
Incidents like this, both to others and to myself, are slowly teaching me to stop assuming the worst in every encounter. This, in turn, increases my self-confidence which further increases my passability because nothing attracts negative attention like insecurity.
My recommendation? Take those ma'ams and run with them! Enjoy them! And just assume those saying that meant exactly what they said.