I've swam with my packer in all manner of water: salt, chlorine, fresh etc. and it hasn't damaged my packer at all.
I use either the GV soft pack or the 3.5" generic packy depending on my mood, and to make sure that chlorine or salt doesn't affect it I always wash with antibacterial soap after swimming, before putting it into my fresh pair or underwear.
I figure that it's probably the best way to ensure longevity of a packer, not to mention keep it as sanitary as possible.
Hope that helped.