Answering the original question, I feel like those words apply for no other reason than those are the words we have to work with, aside from androgynous. There just aren't a lot of words in English to describe a person's gender other than feminine, masculine, and androgynous.
And people are always going to describe others. I'm sure some of us don't want to be described by other people or subject to their choice of descriptions, but that's how we, as humans, understand each other and ourselves.