I also felt very scared at the prospect of changing restrooms around a year ago; basically people read me as female at a distance but once close enough, or at least when I started speaking, I was easily clockable. The turning point for me was when a guy tried showing me the way to the right restroom as I went into a men's one. I remember thinking that if this gentleman read me as female, some less gentle male could too. Or worse, a less gentle male could read me as trans.
The next few months, since I was still not confident that I wouldn't get comments in the women's restroom (or worse, be spoken to and having to answer - I still fear that situation, but my voice is a lot better now; it happened a few days ago and the other woman didn't clock me, even though she didn't see me as she was in a booth), I opted for HC or other solitary restrooms whenever possible. Even though going to the HC restroom felt a bit stigmatizing, to be honest many people on my campus do that simply because the other restrooms are full, or just because they prefer to be alone. There is also no law or regulation forbidding non-handicapped people from using them, where I live.
The next turning point was finding myself in a situation with no available neutral options. Using the men's room was out of the question, so since after as while I couldn't hold my bladder I decided to use the women's room. And it went perfectly well. No glances, no comments. I still chose neutral options whenever possible for a few more months, but after a while, having used the women's restrooms in everything from shopping malls to nightclubs without receiving so much as a glance, I stopped using the HC restrooms altogether. On my campus, some people who recognize me from before have thrown me glances, but they weren't negative, more like a subtle smile saying "Wow, you've changed" - I hadn't seen that student since before I started transitioning. And the people I've hung out with almost daily throughout the whole process, act perfectly natural around me if they see me there too.
This approach worked well for me, but of course it's a compromise - I didn't like using the HC restrooms, but at the time it was more practical and less humiliating than my fear of being clocked in a non-solitary restroom. That said, you have every right to use the restroom you feel comfortable with, and I was probably overly cautious, as several female friends told me that they wouldn't even have given it a second thought if they saw me in the women's room, even if we were strangers. I needed the extra confidence of knowing I pass fairly well before I felt comfortable switching entirely, but for many, using the HC restroom feels too humiliating, and it's perfectly okay to choose either way. I think you should choose what you feel comfortable with.
I'm still in limbo regarding wardrobes, by the way - I train martial arts, and several of the women in my dojo (we train in two different student gyms on campus, so it's not just us in the wardrobes), including my instructor, have said they wouldn't care if I used the women's wardrobe, but since I usually have to switch to another gaff after training because it gets sweaty (I train tucked), I don't feel comfortable changing there as long as I still have that thing between my legs. So in the gym we use for beginners classes, I change in the HC wardrobe, and in the other, since there is no solitary wardrobe, I change in a HC restroom. Not ideal (for one, there are no lockers there, so I have to carry everything I have with me to the dojo), but it works. I'm getting tired of it though, so I have considered just finding some way to change my gaff without exposing my genitals, but since I'll probably have SRS in less than a year, I'll probably just try and be patient and switch wardrobes once the problem is out of the way.
