One thing to watch out for if you're going to pack while swimming is what material your packer is made out of. If it's absorbent at all, i.e. a Mr Limpy, you run the risk of looking like you've had an "accident" when you get out and the rest of you dries faster than your packer. To be on the safeside I'd be tempted not to pack, but go looking for some shorts with a velcro fly, apparently you can't tell the difference with those, and being in cold water would generally make a cis guy's shrink anyway.