I think the folks you are talking about, your masculine female friends, could fit under the trans umbrella if they so chose. Trans is a pretty wide scope of identities. What you are really talking more about are transitioners--people who alter their legal, social, and/or physical state across gender lines.
It is common for binary trans folks to fall into a kind of "either/or" way of thinking about being trans and transition and all that jazz. When the truth is that what we've got is a kind of smushed together collection of tools that we use to try to be more comfortable with our bodies and our lives.
Trans is, outside of the psychiatric diagnosis of Gender Dysphoria (which in no way actually refers to "trans"), an identity label. You are trans if you say you are trans. Therefore someone can't really "incorrectly" think they are trans. In the same way that someone can't "incorrectly" think they are gay. You might realize that what made you label yourself as trans or gay is not true for you: i.e. a person decides not to transition; a person decides they enjoy having sex with people of various genders.
What can, and sadly does, happen is that people can undergo aspects of transition, and then realize that they are unhappy in their altered state. To me, the warning signs for this are fairly straightforward. It is frequently people who are choosing to do a thing in order to achieve a goal that that particular tool can't provide. Often a goal that is far, far too general. In other words, you should undergo phalloplasty, for example, because you want to physically alter your body in that way; not because of any general concept of "becoming/being a man." You should take hormones because you want the specific effects those hormones have on your body. You should dress in a way that is comfortable, both physically and emotionally, for you and that allows you to function in a rather biased society. Obviously, there are compromises that have to be made sometimes, but you get the idea.
The people I see run into trouble tend to be the ones who go for a grand sweeping narrative of who they are and what that means they should do. Each action, each decision, should be independently evaluated on its own merits for the individual in question. Just because there is a preset and expected path for binary transitioners doesn't mean that it is correct for you (or incorrect for you, for that matter).
When people ask how I know I'm trans (this is a rare event), I just say that my body hurt and now it's better. But that is MY answer for my specific situation and refers to how I personally made choices to resolve my ->-bleeped-<-. Your answer is, mostly likely, going to be something different.