I may be weird, but I think it's exactly as indecent for men to go around topless as it is for women. That being said, men don't care about being indecent in general; their sense of decency, their pudic character, is weaker than women's in average. And our societies having been constructed by men and currently adjusting for women and not the contrary, men's standards are the norm unless stated otherwise. I'm not overly frustrated by this fact, but it's how it is.
It is "normal" for a man to remove his shirt when swimming; just "acceptable" not to. And it is "normal" for a woman not to remove her shirt while swimming; just "acceptable" not to. Though as being "less pudic than the norm" is naturally more acceptable than being "more pudic than the norm", it's kind of worse for women.
Don't people who are attracted to men find a topless man just as attractive as a woman if they were attracted to her?