If I were to be in a relationship with a trans/woman, that to me would be a gay relationship. If I were to be in a relationship with a trans/man, that to me would be a straight relationship.
I differentiate to "pan" because I know people who are not on opposite ends of the binary. If I were to be in a relationship with a male-bodied two-spirit/bigender person, it could be offensive to them to describe it as gay or straight. At that point, the relationship would be best described as queer, to me.
Let's take for example a "crossdressing [man] dressed up as a female pirate." How does this person self-identify? If they self-identified a straight cis-male crossdresser, that would be heterosexual attraction for me. If they identified as a gay cis-male crossdresser, a relationship with me could not be truthfully described as gay or straight, but queer.
My own genderqueer offspring (sonter? daun?) is a female-bodied genderfluid person: sometimes they are the woman who is my daughter and sometimes they are the man who is my son.
To me, these things aren't labels but merely the words I use to describe myself. To me, it's just descriptive vocabulary. And also to me, discussions like this in relationships are extremely important. Without communication there can't be a relationship (even if it's just about sex; one still needs to know what one's partner wants and what you want your partner to do for oneself).