I can see that this would feel weird.
I recently had an experience that was weird for almost the opposite reason. I transitioned at a university while I was teaching there part time. I've taught there a couple of terms a year since my transition, and every guy who knew me from before treated me great, as if it were only natural that I'd use the men's room. That was strange at first, but not very. I got used to it. I like it.
Then, this week, I encountered a colleague who most definitely knew me from before. We used to talk now and then, but I haven't run into him in a couple of years because we are always on different schedules. He came into the restroom as I was coming out of the stall, and he didn't even recognize me. Nothing, nada, zip. I wanted to say hi, how are ya, etc., as I do with the other guys. But I didn't want him to suddenly recognize me and think about my past, so I washed my hands in silence.
So my weirdest restroom experience (as a guy) has been when somebody who knew me didn't recognize me and didn't question me.