I don't believe most people would say or do much even if you're not passing. Most people would think "oh, maybe just a mistake" or "ok...whatever." But there is that chance of there being a bad apple, too, and I don't think that being concerned if you're not passing well is completely unfounded. Some people might see us as really butch lesbians and act out in anger/homophobia. I remember my grandfather told me if he saw a woman in the men's bathroom he would call the police and make a big scene, and let me tell you, I believe him on that one. Some people have very strict gender roles ingrained in them, and they will make a way bigger deal out of it than they should. Are they the exception? Yes. But it doesn't make it better if it does happen to you.
I do think that if you're passing though, you should try and use the men's, for your own sake. But if you're like me pre-T (or even now...but pre-T I literally never passed), it may be more comfortable to use the women's, and it might be the right thing to do for you. There has to be a sense of readiness and security, and it comes down to if it would be worse to use the women's and deal with dysphoria, or worse to stress over not passing in the men's and facing harassment. For me, it came down to the dysphoria being worse than the fear, which is why I took the leap and started using the men's.