I'll wager that few people could say what, exactly, they're looking at in deciding gender.
I'll guess that a fair part of it is seeing what they expect. If they expect 'woman', for whatever reason, they'll see that. And if they expect 'man', they'll see that. In terms of things you might do, I'll guess the eyebrows were a good idea. Namely, something to make it easy for others to continue with the expectation of 'woman'. And, from greater distance, some stereotypically woman clothes might help telegraph the idea.