Thank you folks for the replies!
Pica Pica - I think you're right there (and I am going to do more with the exercising). And Cain, you bring up a certainly valid point too. Since I believe that I probably do have some form of intersex going on (long story short, my mom had PCOS before having me and had very high levels of T - so I came out with slightly questionable physical traits, not 100% fully developed internal female parts and hormone levels that I've been told are "not normal" ... I've come to think they are normal for me, but that's another whole story) my body will probably always be a little bit of both. I can handle that, even though if that magic wand showed up I know which way I'd choose. So as far as the gender goes, I think it's male.
But, you also have to account for the fact that for most of my life I've had to live as female. That's where the sociological component comes into play. I could act the part if I really wanted to, but I don't want to – it's not comfortable at all for me, it really does feel like an "act". Even though I feel I'm a bit of an empath at times, interaction between females just confuses me. While I can pick up on some of the nuances of it all and "play along" to an extent, I can't connect with it on their level if that makes any sense, and I usually do end up doing or saying something that makes them all look at me like I'm from outer space.
And basically for my whole life past around age 12 I've been androgynous in appearance (up until that age my parents let me pretty much live as male as far as appearance and activities went). Couldn't tell you how many times I got the "are you a boy or a girl" thing just in school alone. So that brings me to Ativan's equally valid point that normally, if someone were to adopt an androgynous appearance, that's a kind of "transition" as well and I do know there are non-binary people who choose hormones or even some surgeries specifically to gain a middle ground that they're comfortable with. I guess I'm lucky in that I've stood on the middle ground pretty much my whole life and that I'm okay with androgyny (just wish it fell more to the male side for me).
So I think I have kind of a muti-dimensional thing going on where I could technically be placed in a number of different "categories". I would truly love to just say to hell with any categories and just be myself, but these little things keep cropping up that frustrate me.
And Cain, I never really thought in terms of having "social dysphoria" ... just always thought of the physical form of that. That totally makes sense to me putting it that way though as that really is a good description for it. The last couple days I've thought about how I could "sort of" come out to people in a low-level way to try to stop this type of thing from occurring and making me uncomfortable/angry. But then I thought, well, even if I did that they would still consider me female. The very select few (basically 3 friends and the guy I'm married to) that do know, I'm totally comfortable around. So I may just have to come out to some of the "worst offenders" who are still close friends of mine so that this behavior could hopefully be modified.
I think this was helpful. New and different perspectives often help me to sort some of my own stuff out.