The only downside to gender neutral I can think of, is male bodies tend to be capable of exploiting a urinal option with great speed.
If you don't need a toilet and just need a urinal, well not having urinals means you can only use a toilet.
And it is not like you can have some urinals and then a batch of toilets and not offend the females.
Then again, I find speed to only matter in theatres. Because when the film end, suddenly you have a few hundred bladders to cope with. On those occasions, having a male anatomy is actually a bonus. I've seen a process rate in men's rooms that the ladies rooms can't hope to equal.
And all toilets is essentially just a ladies room. If they make them all separate rooms, it is just an inefficient use of building space.