(I think I might be a bit late for this topic, but I haven't heard anyone else saying this)
I wear girl's undershirts (the ones with the 'shelf bra' in them, I feel like the added fabric helps flatten me when I bind) because I still live at home and am not out to my family. I don't have trouble sizing them or anything, and they are tighter than guy's undershirts (from what I've seen, I don't have any firsthand experience) but I've only ever worn them as under layers so it doesn't make much differnce to me. I'm lucky enough to live in a rather cool place, but when it gets hot and I can't wear my binder, I usually layer a couple of sports bras with a girl's undershirt, and with some strategic placement it works alright. I don't have any advice for swimming though. D: