There's nothing wrong swimming in a t-shirt in a common swim pool in either in M or F mode. If you don't want make other ladies uncomfortable, and there's no family changing room, then this is probably your easiest option. I've done it in M mode in our town school swimming pool (I didn't want to show my boobie buttons, or just felt like not showing them) and never had any questions asked.
Although it's 21st century and in most places we are quite acceptable if being mentally strong and have thick skin, and if you have to have swimming for health reasons and the swimming facility can offer you only male changing room, don't take it negative. Just use it, you can be discreet and change behind shower curtain or in the back row of locker room. Think of Europe, medieval times or difficult circumstances when both genders have to survive together.