Hey Jamie... your experience resonates with mine. My first transgender thoughts were that I wanted to be a woman. Sex featured heavily early on. But something about it didn't do it for me. I've lived life on testosterone, and I've now been living on estrogen... and I personally love the change in my libido. Among other things... my overeager libido doesn't color my perceptions so much, and I the only time I find wearing women's clothes to be sexy is when I've got designs on my wife's attention.
I've been taking an observational role with respect to my gender. I have four "gender modes" that I've identified: boy, androgenous, tomboy, and multigender. There might be several multigender modes going on. Perhaps I have a variable amount of masculine and a variable amount of feminine... and they simply aren't related to one another. I'm not really sure, so I just keep observing, and hoping that it settles into a single mode!