My advice would be to:
(1) Buy a size larger than you need
(2) Limit your time wearing it (both hourly and over time generally)
(3) Put some kind of barrier between it and your nipples, as those are probably more sensitive than the rest
I've never personally worn the swimming specific binder, but AndrewB posted a review earlier that makes is seem as though they are just as effective as the normal ones. I think to prevent tissue breakdown, the best thing you could do is get a size larger. I think you'd still need to be very on top of how long you're wearing it in order to not cause any lasting damage.
Another option - google 'modest swimwear'. My family are very devout Christians and don't like to show much skin. They get what are essentially knee length body suits that come up to a t-shirt top. They kind of look like wetsuits, but there is usually a top layer that's more fabric-y to hide the chest and ladybits. Many of them come in neutral coloring, and I'm sure you could make some modifications to it.