I think it's just luck of the draw but you'd be surprised how many people are gendered based on context rather than oughtright appearance. Many people are androgynous and depending upon their mannerisms, clothing style, etc, they are gendered male or female. Of course for as many androgynous looking folk, there are just as many "definitely one or the other."
Now to make context go your way, you can wear makeup to give yourself smoother looking skin and have a feminine haircut. Unless you're a caveman or linebacker (which you are neither) they ought to point you in the right direction. Oh! Eyebrows are another big one. You'd be surprised how much changing eyebrows can change the way a face looks.