-ity and -ism are different ways of forming nouns from adjectives in English (along with other suffixes like -ization, -ification, -ness, -izing, etc). The same root adjective might have a number of different nouns formed from it, each with a different meaning; the difference could be subtle or dramatic.
In the case of "sexual," the noun form "sexuality" relates to areas of sexual activity, interest, and orientation. So "homosexuality" refers to activity with, interest in, or orientation toward the same sex. Someone familiar with these connotations could read "transsexuality" to be an orientation label as well, perhaps assuming that it referred to a sexual interest in trans people, or sexual arousal from crossing gender lines. "Transsexualism" avoids that potential confusion. Some people nevertheless use "transsexuality" because it sounds more familiar and natural.
