First of all, don't worry about Dick Floyd the Tennessee Republican who threatened to stomp on transgender people in the bathroom. He is an idiot and would never do such a thing in real life. Also, you don't live in his state, so don't worry about him coming after you

Seriously, if you are passing so well, I doubt people would really think twice about you using the ladies room. Even if you aren't wearing a dress and have a cap on, it really doesn't matter. It's not like there is a bouncer at the door saying "girl what are you wearing". People have different styles. If you pass and your clothing is at least unisex, there really isn't much of an issue. As for the beard shadow, it's not like people are going to stare intensely at your face as you enter the restroom. If you manage it well, who is really going to notice?
However, I do understand your concern. Luckily for me this will never be an issue. I have a major phobia of public bathrooms and the uncleanliness of it all that I ensure to hold it in as best I can. There are fewer places more disgusting than public men's rooms.