It's big for me but only recently. I don't trust women. I worry they'll either miss the penis and leave me for that, or they'll secretly fetishize my hated parts and use me as liberal brownie points ("oh, I dated a transgender lol I'm so open minded").
But it would never be enough to make me detransition. You can be surrounded by people and loved, but still feel completely and utterly alone. And that's how I felt as a woman and as a non-passing trans man. People found me attractive and I had lots of people who thought I was "cool." But I felt so alone because I knew despite their best intentions they couldn't see past the outside. I ditched almost everyone except the few true blue friends who did all they could to ignore the lies of my outward appearance...and frankly, I am better off without everyone I walked away from because although I wasn't physically alone those people made me feel alone, trapped in my skin. Detransitioning might make it easier to get a date, but I would know I wasn't being seen or loved for myself and there would be no point.
Besides that, real love is not so fickle. Being trans might make casual dating harder, because unfortunately, I place very high expectations on others in terms of understanding my transition. I've been hurt too many times by people who only half ass know what they're talking about, even other trans people have done this. So when someone doesn't "get it" enough for me, I don't care to continue the relationship/friendship, because I've been through too much to do that again. But I know that someone who REALLY loves me, when they find out about my past - they're going to find a way to accept me exactly as I am, without fetishizing me, making assumptions about me, etc because they're going to see that in spite of my challenges I am a man worth loving. Who respects how I feel about myself and lets me tell her who I am, not the other way around. I'll find that woman one day, providing that I can allow myself to take a risk and trust her.