There is also a pure political correctness component to it. I've read a number of places people(particularly but hardly limited to the more politically radical and outspoken that tend to shape terminology) have stated they consider the word to be derogatory due to things such as external labeling and the association of Transsexual with the porn industry. Not to go down the PC word choice rabbit hole in full, but it is hard not to mention it given the topic. Essentially, it is common practice to shape dialogue based around approved words and phrasing. Sometimes it is entirely legitimate to avoid blatantly hurtful dialogue (particularly with certain racial slurs), while other times it is simply an attempt to create a political rallying point (including the dismissal of medical terms) by attempting to use previously uncontested terminology to artificially recreate the more legitimate instances of politically correct word policing. In this case, it's somewhere in-between probably, as the word has never risen to the level of certain well known racial slurs, but at the same time a legitimate case can be made it muddies the discussion by associating gender identity with sex, even if in a purely happenstance manner based on the dual meanings of the word sex itself (biological sex of an individual, versus the action of procreation).
I personally feel like it's best not to consider it offensive, because I've never seen anyone use it as an intentionally derogatory term, but that using the gender based labeling is just easier in a non-medical or gender layman's setting. (In other words, it's fine to use the term when people understand the real definition, but it's better avoided when the audience may not.)
I worry about avoiding it based on stuff like porn however, because that is a never ending cycle where the industry will just switch to the new words, then those words will become non grata, then any new words chosen will eventually be used by porn, making those in turn verbotten, etc., etc.
