I personally would not use gender-neutral bathrooms when they exist in addition to traditional m/f bathrooms unless there was a long line and a sense of urgency. It's nice they do exist, as I think a lot of non-binary people would benefit, as would families with small children. Sadly, it's probably best they exist for those of us who don't blend in well.
For me, if I were to use the neutral bathroom I might as well just put on a flashing neon sign that says "I'm trans- look at me!" I try not to draw attention to myself, as I am quite tall and have a large frame. I have never once had a problem using the ladies' room, as I am "plausibly female" enough these days to get away with it. I also would never feel safe again in a mens' room. I've seen the way GGs often get treated in them, and it churns my stomach. (Now why a GG would want to wade in the overflow in the first place is beyond me.) Presenting female in a mens room outside of a gayborhood could mean very bad things as well. I don't go there- ever.
One thing that happened to me recently was going to the ladies' room after a concert, and while standing in the long line, a woman screamed, "Oh my God, there's a MAN in here! GET THE F OUT or I'm calling the cops you effing pervert!" Turns out I was not the individual in question. It was clearly a GG with a binder and a bit of dark unshaven facial hair. *facepalm*
Perhaps the best thing is just to have a row of individual unisex toilet stalls with sinks and mirrors? I've seen this setup in recently constructed buildings and it seems to work just fine.