If I do swim, it's in a non-public pool or the ocean and wear an old t-shirt/gym shorts. Thanks to dysphoria though, I've avoided swimming for about two years now. Sucks cause I actually enjoyed swimming beforehand.
A side note, a few years back I went to the beach with my family and stayed at a hotel with a pool. My mom asked the front desk if I would be allowed to swim with my t-shirt/shorts on because of my anxiety. They said there was no problem and put a note in for the lifeguard, and said if the lifeguard complained to tell them to talk to the front desk. I'm the US and this was just one hotel, but if you really wanted to swim there isn't any harm in asking and seeing what they have to say.