This is something I find utterly fascinating, and I think it's one of those questions that cuts very deep into the problem how the brain works.
I could tell you the specific things we look for -- particular shades of coloring, angles of bones, patterns of shadow, slight cues in the lips, the nose, the eyebrow ridge, the forehead, the chin, the jaw, plus various other cultural cues -- but you probably know that. I think most people can't really put a finger on these differences, but a lot of trans people are much more aware, since it's so important if you're trans. Either way, it's amazing how automatic and obvious it is. It's so reliable and such a common part of everyday experience that it really upsets up when it doesn't work.
I think a huge part of it is instinctual. Consider babies -- they just love looking at faces. There has to be some instinctual response that triggers that behavior. I think that this instinct might be related to how gender identity comes about -- that is, just as we are programmed to recognize human faces, so we are programmed to recognize faces of our own gender. Whatever else is going on in the brain that might differ between men and women, I think what decides your "brain sex" is what type of face you are programmed to recognize as "like you," regardless of what you look like yourself.
Well, that's my tentative theory for now, at least.