Well, it is more complex than that, MUCH more complex actually.
For starters, gender is a word that comes from linguistics. In Spanish, for example, banco (stool) is a word with masculine gender, while silla (chair) has feminine gender. No penises or vaginas involved. That's why the word was brought over to social sciences, to talk about something that doesn't involve penises, vaginas, hormones, gonads, etc.
And yes, it is a social construct. There's nothing wrong with that. Ying and Yang, feminine and masculine, have attributes that are different from the feminine and masculine archetypes in other civilisations, like the native peoples of America.
Does that mean we don't change our sex? Well, no, it's not so simple as I said. We have several sexes. Usually they match, they're all male or all female (noticed how I didn't use those words when talking about gender? Those words are strictly about sex, for gender we use masculine and feminine, man and woman, boy and girl), but not always, not even for cis people.
What are all these sexes? Well, I'm actually trying to find this text where "gender" was first used to talk about people, I need it for something I'm writing, so hopefully tonight I can answer properly, but there are like four or five. Off the top of my head I remember chromosomal sex, hormonal sex and genital sex. So yeah, we definitely change some of them.