I'll place here as a disclaimer that I'm pretty much strictly attracted to men, cis or trans. I think my aversion comes not from the possible reasons mentioned by the OP but somewhat from ftmax's reasoning (though I didn't realise it until he pointed it out) plus a little bit of my own preferences. I have long since dissociated genitalia with gender/sex, so that part doesn't really bother me, if I'm considering a guy at face-value. That being said, I am attracted to the overall 'image' of a man, so there might be some people that disagree with me, but to me if a guy, cis or trans, doesn't fit my image of an attractive manly man, I'd probably be less likely to date him; it's one of the reasons I'm not generally interested in non-binary/gender-fluid peeps, because I'd only be physically attracted to their more masculine side, and that's definitely not fair to them.
I'm also horrible at helping people with their baggage, esp. when it's due to mental illness; my best friend has depression and anxiety and I know just trying to help her during her lows drains me enough, let alone if I was dating someone in a similar situation. I can handle general baggage, bad days, etc., but I have a hard enough time dealing with my own issues with being trans; I'm going about as stealth as I can, and plan to continue that through college (and thereafter) so dating someone that might try to bring me out of my little stealth world, mentally or socially, would be a deal breaker.
As a last note, I also get a lot of guilt when I experience certain things that other trans guys I know have not, such as HRT and its developmental milestones—such as facial hair—or surgery approval, or even legal milestones like name and gender change. I wouldn't want to be in a relationship where I feel like I have to set my elation aside, even if it's only me that feels that I have to.