I think the majority of people are generally mis-informed on what it means to be a trans-man,as you put it. If someone who is sexually attracted to female body parts reads being a trans man as "a man with a female body" to them it is still fair game? ( Not that we are animals to be hunted or anything but you get my drift) while offensive I find it hard to get mad at those people who assume this because it is purely ignorance. On the other hand if those same people ARE properly educated on trans-men and still fail to acknowledge that they are male I would find that HIGHLY offensive. But, I do disagree on your "fully heterosexual" "fully homosexual" argument.
What if you identified as a straight male met someone on-line whom you fell in love with but they turned out to be a male, would that make you a homosexual? or vice-versa I don't think so . As others before me have said, gender and sexuality are more then what can be defined into words, labels and categories. Even though we are taught to believe otherwise. I love my wife, would that love just disappear if she came out to me as a transsexual. No not at all, but that doesn't make me any less of a heterosexual man. I can see a man walking down the street and appreciate his looks without calling into question my sexual identity. Break outside of the box man