Well, the first two therapists I ever saw refused to treat me properly because they thought I was being a big baby for being so stressed out about the influence of fundamentalist religion in my life, so I'm plenty familiar with terrible therapists. If I've find one who's not abhorrent, I'd like to just stick with them.
I'm not sure what a "gender therapist" is actually going to do that a standard therapist who I like can't give me, the therapist I went to who actually was experienced with transgender patients just kept making stereotypical stock suggestions instead of paying attention to my individual needs, and was way more pushy with transition as the only solution than I was comfortable with at the time. If I'm going to transition, it's going to be because I choose to do it on my own, not because someone who's spent all of two hours studying me ordered me too.