Look, no one will care. I went swimming in a local lake the other week in a T shirt while everyone else was in shorts/bikinis, no one even batted an eyelid. It's a big thing for us, because we feel like it's so obvious why we're wearing one, but guys wearing T shirts in pools is hardly groundbreaking. People will only start to wonder if you get anxious and defensive about it. If someone asks, tell them protection from the sun, and that's it. No one cares.