In the trans community I hear a lot about passing, especially for trans women. I am fortunate enough to be at the point where I can pass. One of the reasons why I think many stress a lot about passing because for some it can definitely be a safety issue. It is horrible that we are more prone to harassment and violence by the public at large. Often times it is demanded that our bodies be in the public domain, very private things you would never ask cis people are directed towards us. Many questions asked aren't meant to hurt or demean us but it is very annoying to hear such questions. Unfortunately we live in a society that doesn't accept or tolerates femininity from MAAB people ranging from cis guys to trans women. It's unfortunate and sucks that we are looked down so much from so many. For me that is one reason I want to pass, so I can live at peace in public, most people are nice or don't care but all it takes is one person to ruin everything. I admit it is one reason among many and not everyone feels the same.
I have two reasons for concern about passing. First it is a private matter. Much like what you have already stated, I don't want to have to worry about the questions and negativity that comes with getting clocked. The second is that male pronouns feel like a huge slap in the face and there are some out there who, whether intentional or unintentional, will always use those on someone they assume to be male.
I feel the same way as you about this, though I really don't feel like I'm able to pass which is why I take my time with transitioning. While I completely respect those that are out and proud, it's really not for me. Still caring about passing can have it's down sides. When you feel so strongly about it and you don't believe you can, it can hold you back in many ways. And I suspect many of those that do pass may still remain insecure with their ability to pass and their appearance if it's given such a huge important. Sometimes it's very constricting and hard to handle. So, I am with you, though I realize that cariing about it too much probably has been hurtful to me in some ways.