I'm obviously not going to use the male restrooms, and I'm not going to use a neutral one if the women's is available but if the only available option is gender neutral then I can't see what the fuss is... Unless of course some man has used it before and has left the seat up and peed all over the floor, or worse, left the seat down and peed all over the seat.
Before I finally decided to admit that I was trans gender, all the restrooms at my workplace were unisex single stall and that was great as it didn't make me feel separated from the women I worked with. I was enough for me. The day they made one of the restrooms female only was the day I had my straw that broke the camel's back dysphoria meltdown.
Neutral restrooms options can, I believe, serve a very useful purpose to trans people who have not transitioned and/or are trying to manage their dysphoria. For them to be proposed as the "solution" for those who have transitioned to use is of course insulting but that doesn't sound like what was happening at your therapists, it was applied across all genders, cis and trans equally.
The building in which I work recently turned their M/F ground floor toilets into unisex, but so they could also add an accessible toilet for disability use. I haven't needed to use the restroom when I've been on the ground floor yet but when I do I'm not going to go all the way up to the next floor just to use a gendered stall.