Yes, the sexually dimorphic nucleus in the hypothalamus is only present in biological males, and is at least partially responsible for directing sexual behaviour towards archetypally 'female' attributes. People with lesions to this nucleus sometimes became bisexual.
I haven't seen anything which gives a neurological basis for gender in general aside from in terms of the empathising-systemising spectrum. But since we can have primarily empathising men and primarily systemising women, surely even neurological sexual dimorphism can't determine gender?
What I'm not sure of is, since you can have women who are completely 'masculine' in everything they do, and men who are completely 'feminine' in everything they do (i.e. people who would not blink twice if they were born in as the other sex, but still prefer the one they are in), can gender ever be accurately defined as anytrhing more than your preference for a certain kind of anatomy?
I'm fine with this being the case, by the way, because in my eyes it allows gender to be a primarily social concept, but still explains transsexualism.