YES!
I am having this same problem. I really don't know why? I don't think I look to much like a girl when I'm not trying too. I have had men come in and walk out, thinking there in the wrong bathroom. I have no choice but to use the men's room as I am not presenting female AT ALL!
I usually get looks and Mam's when my hair is up, but I try to wear it down and that doesn't work always either. Yes I could look grungier but I definitely think I don't look female.
I posted once, in a thread "how you can't pass a male any more". I posted that no matter how well you pass when presenting I still think one can pass as male. I am now finding out that it, IS, getting somewhat hard to pass as male, even if I do look grungy. That doesn't mean I still couldn't pass as male, it just means I actually have to try a little to do so.
Yes! I'm happy about this new found revelation, but its also complicated when I am not out yet and present male most of the time.
Its especially awkward when a cashier Mam's me and I presenting Male, do I correct them or do I just go with the flow. I have been just going with the flow, but its REALLY awkward when my children are present.
Then I also get to thinking, geez I wonder if some of these people think I'm just a poorly dressed cross dresser.

Sometimes I think it would just be easier to come out and live full time.

Shelly