In my opinion, you can in no way base bathroom rights on attire because there is no standard for attire, let alone a standard for masculinity or femininity. That would be an impossible guideline to follow, let alone enforce.
There is zero statistical danger of perverts doing anything to anyone in bathrooms. It never happens and they are relatively assault unfriendly places.
Gender markers on bathroom doors where never meant to be legally binding barriers. Trying to make them that, is a useless exercise in effort and cost, for no return in safety. Yes more unisex and single use bathrooms helps this all for sure, but that alone is not the answer.
The answer is move on from this political culture issue to something that actually protects people. Maybe campus safety, maybe more education, maybe better family counseling. These are some things that could make people safer.