This is something I certainly notice as a privledge as someone who passes and has my documentation reflect my actual gender and name. When I talk to people even those who know I am trans they tend to be very adamant that people jump thru lots of legal hoops in order to use the right restroom, such as gender markers on ID or even a "bonafide feminine presentation" as one put it. But they never say I shouldn't use the same restroom as them....I find that sort of doublethink to be very dangerous.
I know I for one don't want any authority saying what is and is not proper clothing to use to pee in the restroom at the department store, and I don't think it is very reasonable to expect people to have all of their paperwork in order either; it just isn't practical for a great many transwomen as many states and localities don't make it easy.