I, too, have issues with transgender terminology. They usually arise from inconsistent meanings associated with labels that lead to miscommunication. For example, some people use the term "gender identity" to denote an intrinsic, essentialist quality of an individual which is present at birth and unchanging, while others see it as a characteristic that can be donned and doffed as circumstances call.
I'm in the latter camp because, to my way of thinking, gender identity is not a one-way street. My gender identity is both my own sense of gender and how other people see me. If there's agreement, there's harmony in gender relevant social interactions, and if not, it'll depend on the situation.
For most of my life I identified as a boy/man. Because I was born male-bodied, there was no disagreement over where I fit within the gendered social structure. That gender identity, however, did not agree with my immutable, neurological (brain) sex which biological processes rendered in the womb.
When I decided to change my gender identity to match my neurological sex, it clashed with the gender identity that others perceived. People still identified me as being male. It wasn't until I changed that perception so that my sex and gender merged into a consistent, congruent state of being that gender dysphoria was relieved.
So I see a person's gender identity as being both chosen and bestowed. My transition has been a long, difficult pursuit of a gender identity that matches my neurological sex; that is, to have others acknowledge and accept the gender identity I've chosen for myself so that I can live free of chronic gender anxiety and distress.
The controversy over access to sex-segregated spaces disappears for those whose gender identity is not in question, that is, when a person 'passes' as the gender they've chosen to present as. It's only when other's contest your chosen gender identity that the potential for confrontation occurs.
I seriously doubt that the linking one's gender identity to their sex will ever become irrelevant. Sex drives all human life, and gender identity is its proxy.