Well I am not only past transition but I'm 25 years past SRS, and the fact is that in all that time I have seldom if ever be "read" by anyone who does not have inside knowledge.
Looking at my rather average and uspectacular photo you may wonder if I am making that up. But actually I'm not, because you see the secret of passing or not isn't so much about looking perfect, it's about attitude and self belief.
If you project someone at ease with yourself, if you genuinely do not expect to get read (and so unconsciously project that there is nothing to read), and if you don't really care anyway the chances are that, if you look even halfway female, people will take you at face value because it's much easier than trying to "read" something which from their perspective they don't know is there to "read" anyway.
My secret is that I don't try to hide my flaws, I seldom wear much makeup, I dress in a fairly middle of the road way, I am just myself. I don't TRY to pass at all, so paradoxically I just do, because people take me as a fairly ordinary down to earth slightly gender neutral woman.
Although I am technically Bi I am frequently read as a lesbian, by other lesbians including those who normally wouldn't accept a postop transwoman as a valid lesbian. But far from getting upset and pointing out that I am actually bi I take this as a huge compliment.
So far from criticising you I personally would commend you for your realistic attitude. It doesn't matter at all what others think, and once you get past the point of thinking that it does, that, in my experience, is when you start to pass pretty well.
Bottom line if someone called me sir, I would just laugh smile sweetly, maybe unbutton my coat a little and wait for the inevitable "oh I'm terribly sorry ma'am". Natal women don't get upset, it's the getting upset and maybe even correcting someone which is actually one of the real giveaways.
So I think you are absolutely right. Stop worrying and just enjoy being yourself because in the main people seem to give a hard time to those who seem unsure or over eager to impress.

"So frankly my dear I don't give a damn!"