Binary: 2. relating to, composed of, or involving two things
That is my understanding of the term when applied to trans people or gender. That is, that there are only two choices, male or female. Many of us, however, clearly feel and see a wealth of choices, presentations and a real history that includes a spectrum. A spectrum suggest relative degrees or components of gender identity with the androgyne accepting and even celebrating combinations or ambiguity.
Women wearing pants in the 1930s was a scandal as pants were only for men. Now, in the modern western world, pants are no longer a big deal and everybody wears them. I think that is happening with gender too. At some point in transworld the choice was, are you becoming a man or woman? More of us now see the shades of gray and the pleasures of fluidity. We have more options from our palette that helps us paint the rainbow.
To me, gender is more of cultural construct while sex refers to our biological reality. My sex was determined as male at birth but I never felt like I fit the current cultural definitions of boy or man. No doubt, a penis is a male reproductive organ while a nurse was often seen as a female role in the past. Binary choices are just two limiting:-)