In the City of Portland, Oregon, it is legally acceptable for men and women equally to go topless without a care, if that's what they want to do. From a social perspective (getting along with friends, family and neighbors), your best bet is to learn from experience. Depending on the time and place, if you're sensitive to the discomfort of yourself and others, then you will have the answer to your question from direct experience. All else is simply opinion and worth exactly what you paid for it.
Hugs!
Miharu