I think voice is just about one of the hardest things to change, and that's just speaking, singing is even harder. It's so instinctive - babies will sing long before they talk - your born with that ability and that makes it very hard to unlearn and relearn because you don't remember having to learn it in the first place.
And the pitch thing is not the deal, as Aaron pointed out you can still tell the difference, and singing is jam packed with gender clues that you can pretty much hit high 90 level % with people listening and making an ID. And none of that is even a conscious thought process, that ability to distinguish and differentiate is also innate. It also often bears little relationship to the speaking voice or even the body, its weird. Another Aaron, Aaron Neville is a pretty big guy, large arms and shoulders, tall, packing some heft, yet when he sings he has this amazing falsetto, but you can still tell its male, even being that high. Likewise Laura Branigan (Self-Control, Gloria) sings in a very male range (which is way so many male comedian parodied her) but you always know its a girl, not a guy singing.
