I've had a boyfriend say it before. Actually, I'm a very sexual person but in certain situations I'm not. Like when a guy tries to seduce me as if I were some kind of woman, I just get turned off and don't want it. I don't mind being seduced, just not like a woman.
When you're trans, you usually don't approach sex the way people think you're 'supposed to', so their conclusion is that you must be asexual or otherwise abnormal (to them).
Don't listen to them. You know who you are and whether you're asexual or not.