I think maybe some trans people have trouble with the non-binary thing (just like people outside the community). Afterall, we are all raised in environments where things are binary... it seems "right".
So, when they finally decide that being at one side of the binary isn't for them and they start to transition... in their mind, the only possible "right" thing is to go to the other binary.
If someone is attracted to them as a girl because they are hot or have girl parts... that is totally fine. That is what is expected, that is "normal", that is part of what being a binary is (and attraction / sex in general).
But, if they are still in the middle... and someone is attracted to them for what they are at that point... they don't think it is right. They feel their bodies are still broken and not in the zone of "normal" yet.
A lot really are binary... but, I think maybe some just have trouble with being different