I think your post is an interesting one. I have asked myself this question a few times.
You know, the thing is for me, putting myself in a box called 'androgyne' is very dualistic for me, just like many things in my life are very dualistic, like my gender feeling.
You ask yourself 'what does it matter'. You can only give yourself the answer that is right for you.
For me, it is like this;
I have always felt somewhere in between. I never felt like a girl when I was a child, and I never feel like a woman now I am grown up. But I also never felt like a guy. And I also always felt girlish and womanish, and boyish and masculine. So it is neither and both. Always this dualism.
I have always felt being from the two genders .
This made me unhappy and confused for many times. It made me think there was something really wrong with me. I could feel jealousy towards 'real' woman, because they had something that I never understood. I didn't belong to them and I still don't, and at the same time I don't belong to men. I could feel jealousy for that too.
I thought I was some kind of a freak. Something must be really wrong with me. Maybe my ->-bleeped-<-ed-up childhood was an influence. But now I know better.
Yes, I do have a ->-bleeped-<-ed up childhood and every reason why I can distance myself from being a woman. But unfortunately as that is, that is not why I have those feelings of androginity. And in my heart I have always known that. It just made everything more complicated, because it was a nice excuse in a way.
The truth is that I truly AM androgyne. Now, I can say that with pride and without feelings of shame or confusion.
I have spend many years in dealing with my past, and I did a great job in it. I became a strong, self-confident and self conscious person. But those androgyne feelings never went away.
One of my friends has a child. He is born a boy, but clearly he is not a boy. This is the only transgender child I know. Other friends of mine also have children. They are clearly from the gender they are born with. To see this, and see such gender-specific behaviour when the children could hardly talk, made me realise even more that what I have always felt is right for me. I am not a freak, and definitely not mentally disturbed, I was always right.
I actually do think this is not a mental thing. I think it has got something to do with my hormones and the way my brains work.
I was proud of my androgynity for the first time when I was 14. I tattooed a symbol of androginity on my own leg. The feeling I had when I discovered that symbol was a feeling of freedom. Finally I saw something that expressed the way I felt.
I let the label go. Until a few years ago. It was just too much of an issue for me, this whole gender-feeling I had/have.
I have a love/hate relationship with a label like that. It gives me freedom, and puts me behind bars at the same time. In a way I declare myself abnormal using that label, especially to people who feel like the gender they are. They will never be able to fully understand it. I also declare myself normal with it, because this way I could find more people who feel the same thing. They (you) help me to accept and understand myself better.
I am a bit cautious now with who I tell what I feel. As a 'biological (?) woman' it is easy to dress androgyne without getting strange looks. And I don't feel the need to express myself to the point that I want to look like a true man. I am not into wearing the moustache, allthough I have to admit I can find many dragkings extremely sexy and sometimes I wish to look like that myself sometimes.
So, in that way it is mental, it is the way I feel and I don't feel the urge a lot to prove this to other people.
For a while I did, and I told many people about my feelings. Most friends of mine know about this. I think most accept it but just don't get it. That also confused me a lot to be honest. I AM a normal person, I concider androgynes or other people between the genders to be absolutely normal. But most people don't. So I was stigmatising myself, and feeling the frustration from that because I do think that we should ALL be considered normal people.
Can you still follow my story?

Anyway, why does it matter,
it matter because it makes me accept myself and find people like here to help me understand myself
it doesn't matter ->-bleeped-<- because I am who I am and I don't need to be boxed for it.
Like my boyfriend always says: What does it all matter what label you are, just feel yourself and be yourself and live yourself!
Besides that I díd find recognition (and thus freedom) by labeling myself, he is absolutely right!

.....I have to add one more thing... about the clothes I wear, I have to admit that when I feel more masculine (I am kinda fluid and can feel from pretty femimine to pretty masculine) I do feel the need to look less like a woman. In that moments I want to shave my hair off and dress much more like a man.
Because I am afraid that I might regret the hairthing, I still don't have the guts to shave it all off.... but actually recently I am thinking about that strongly again!.....