(Note: I'm probably what would be called demisexual so I don't really understand the whole attraction properly first hand.)
To me, if someone's going around looking for trans people to date then that's just plain weird in my eyes. I know that not everyone feels the same, but this is a medical condition to me, and I think most people would say it wasn't normal, or that it was fetishising to go around seeking out diabetics or people with liver disease. You could justify that they have a different out look on life because they have to be wary and careful on how they life their life, the same way as some people have been saying that it's okay to justify seeking out trans men because they were socialised as a female in the past. I can guarantee that not everyone who has the same medical condition has the same outlook on life as everyone else, just as not all trans men have the same outlook or views on women as each other (anyone remember all that stuff going down last year about the guy who was a trans representative and had apparently done sexually assaulted at lot of his partners? I forget his name.) I also take slight offense at the statement that all trans men have taken something from being "socialised as female", really? I've always made sure that the guys realised that I was just as good, and just as manly as them, never really befriended females, and spent a lot of time when I did try to offending them because I have male humour. I find it hard to understand women's fears about walking places in the dark, about strange men etc because I feel like as a female bodied person I should at least be able to relate to that, or have it ingrained into me by society that I need to watch out, but it never happened. Once I was worried walking down a strange alley in town, but that was because I look about twelve, and clearly out of place in a town.
I see it different as wanting a cis male or a cis female because I don't see me as another gender. I am male, my body isn't, but I will do everything I can to sort that out. I probably won't go for bottom surgery unless it advances more, or my dysphoria increases down there, but I am still male, and I don't want anyone to be interested in me because of having an extra hole. I don't even think that someone only interested in vagina would find a post-T, pre-op set of genitalia acceptable because it's not your typical cis-female's, it's not a cis-males either, but closer to that in terms of smell, taste and function (apparently, obviously I don't know yet.) Also, people do have sexual orientations, and no matter how much you love and care about someone's brain, their body does play a role in whether you're going to work out for good or not. I get nothing out of being with men, I don't feel anything kissing them, or having sex with them, not even the close intimate feeling, and the person I discovered that with was so close to me spoke almost non-stop for 8 months, and I did love their mind, it just didn't work with the body. Sexual orientation is more than a fetish.
My ex is now with another trans man, at first I did think that they were being fetishising, but now I don't think that's the case. You don't go running to France without a valid passport and sneaking through immigration on the basis of a fetish, there are plenty of other trans men in the country which would do just as well if that was the case. I didn't even know I was trans at the time they fell for me. Some people probably do end up with a disproportionate number of trans people in their relationship history, but that doesn't mean that they all have fetishes, after all some of them probably get involved in the community like my ex did (and then realised they were GQ/MTF but that's a different matter).
Obviously some people do have fetishes for trans people, men and women, but I don't think it's fair to say that they should be all that we should be in a relationship with if we want to expect it to last. I'm not the sum of my parts, I can't see why anyone would think that I was a good candidate for a partner just because what I have and what I don't have attached to me, and I sure as hell can't see myself ever being attracted to someone who only paid attention to me because of the fact that I was born like this. I'm not saying that attraction is wrong, I find my girlfriend very attractive, and she happens to have similar physical features to the woman who made me realise that I liked women; dark hair, grey/blue eyes, shortish, poor eyesight and neither stick thin or fat. I didn't want to be with her because she needs glasses, I'm with her for who she is, not her features, I've just happened to fall for two people who shared that characteristic in my life, it wasn't a conscious decision.
Actually, I think that's where I'd say that fetish starts and harmless preference ends, when someone is consciously seeks out someone because of a particular aspect, fat, trans, disabled, I have no idea what body-things people fetish over.