I don't think it has anything to do with passing or not passing, it has to deal with how you identify.
Okay, I go out in what some people might call "guy mode," wearing unisex t-shirts, no makeup, my hair tied back, and I do indeed get gendered male a lot when I'm dressed like that. (It's probably about 50/50 male/female.)
The thing is, though, I'm not doing it to be a guy. I'm doing it because I'm feeling too lazy to girl it up. Girls do this too... it's called going shopping in your pajamas because you're too lazy to get dressed. Or suppose a woman wore male clothes... it would be called being butch, not being male.
And again, that's the thing. I don't WANT to be male when I do those things. Really I'm just going out in loose-fitting clothes with no effort put into it, and the fact that I'm gendered male instead of just being seen as a girl who's being lazy or butching it up has nothing to do with my female identity changing, it's just a biproduct of being a tall large-built trans woman instead of a cis woman. It doesn't make me any less of a girl just because I'm not getting gendered female, because my identity doesn't change.
If someone was going out in male clothing with the intent of being gendered male, then maybe you could say that they're a bit non-binary, (which is fine, because honestly I don't think a true binary really exists. I'm sure a lot of cis girls and guys would dress as the opposite just for the hell of it if they had the same option we do of being read as either gender depending on how they dress,) but if you're just going out in certain clothes with the intent of being read as a butch woman or a femme man, then it doesn't really make you non-binary, just maybe a bit gender-nonconforming.