Oddly enough... My first experiments with cross-dressing (outside of dress-up with the other girls in kindergarten before they rejected me) was when I was 14 and 15. I had a good friend who would let me dress in her clothes around her house. The whole family accepted me for me and I didn't trust anyone else at that point. Somehow they all knew, while I was still kinda clueless. There was nothing sexual about it at all while I was there.
At the same time I was ditching school pretty much all the time because I got jumped by a group of jocks for wearing a skirt to school one day for a drama project.
I stopped going to school because I was terrified they'd kill me if I was caught there so I would stay at home while my mom was at work and steal time wearing her clothes... Which were a turn on at that point, I think because I was alone.
By the end of my freshman year my mom asked me about the cross-dressing and I freaked out, became ashamed, became really angry at everything, and started to hide. Pushed everything down and stopped dressing... From that point on, however, my imagination regarding being female was almost solely restricted to sexual fantasy (usually auto-abusive in nature thanks to a history of abuse in my past). It was an easy way, I think, for me to assume it was just a kink and the result of unresolved issues concerning the abuse I suffered. Great freaking excuse, actually. Ugh.

I finally hit the wall and took a good long look at my past fairly recently and figured it out. I'm trans. And Poof!
I have no more random thoughts about sex at all. The desire is gone. I CAN be aroused if I try. But I have to try. It's weird. My feelings of being female are really, really strong and constant... And there's nothing sexual about those feelings at all.
Everyone is different I suppose. But to your question: I think it may matter some because who we are as women or men
can be intertwined into our sexual attractions. But it doesn't have to be. I used to conflate the two but now I don't. Who knows how all that would work for you? You would be the only one who can answer that.
Just my two cents, for whatever they are worth. Take care! <3