"Passing", to me isn't a matter of how comfortable I am. It's more a matter of how safe I am.
Growing up, I learned to "pass" as male, to avoid being targeted (and beaten) in the schoolyard. Eventually I developed a male persona, a collection of behaviors and habits I could run to more or less fit in with the male subculture that let me avoid triggering their "Outsider!" response. I could hide behind that persona, safe from abuse, and isolated from any real social interactions. Safe, but fundamentally damaged; that was me.
I'm dismantling the male persona now with some help, and letting myself out. I very likely don't "pass" as male or female at this point, but as a better defense mechanism, I try to avoid being noticed in environments where I could provoke an aggressive response to my very existence. I am very fortunate to live in a region that is relatively trans-friendly.
I don't really want to be concerned about the perception of others beyond the risk of aggressive responses to the sight of me. If someone else suffers from a transphobic reaction to my existence, that's their problem (unless they go for a physical attack, of course).