I think the actual answer to the question is mostly just a cultural and psychological thing. Many African or tribal cultures go around almost completely naked. But in our western cultures such things are not considered to be ok. It's just the way our current society works. And I personally think maybe our culture came to that based on growing from a culture that was minoratizing women and of a higher standard sophistication. I think it works similarly to public displays of affection where too many people don't want to see it in public and ask that people keep it to themselves. So even though there are people who say "who cares we are all human," the vast majority of this particular society doesn't think that way so in order to get around that you either need to move to a different culture where it's tolerated, or wait/help influence society's beliefs to change on the matter.
On a side note, there's also the argument that men's brains are hardwired to become sexually aroused on the site of women's breasts where women don't really get that way as much in seeing men's breasts. But as we know there exist cultures who live in the nude despite this, so in theory it can happen, but it's gonna be one hell of a long time before I see a country like the US decide to do that too.
