Sex drive is largerly psychological. I think most people who have a partner they want to have sex with and are relatively fine with their genitals retain a sexual activity.
As for the intensity and speed of psychological effects on HRT, I think it all comes down to how much T used to negatively affect the person. I think people who best managed to be themselves prior to hormones, despite male hormone levels, feel the least effects.
Also, I think there are personality traits/behaviours that are brought/intensified by T, such as aggressivity, for example. Perhaps one whose "core personality", regardless of gender, includes some of those traits (someone very assertive and straightforward in this example, I guess), or someone who values such behaviours (very liberal people in this example, maybe), would be less negatively affected by testosterone than someone who isn't really aggressive and hates herself for feeling angry. Thus, the personality changes once on HRT would be less apparent to the former than the latter.
And, hm, there's also the sex drive reduction. I think almost all of us lose some amount of sex drive in a purely physical way. That could be a huge encouragement for someone who hates her genitals or sex a lot, and a huge discouragement for someone who has a very active sex life and little genital dysphoria. Hence, the former would feel more happy and free, whilst the latter wouldn't feel that at all. And more happiness tends to make a person's personality change, or at least appear to.
