I wear a harness and trunks. Both of the pairs I have that I swim in have the same type of liner you mentioned, but it doesn't feel secure enough for me by itself if I'm moving around a lot. If you're just hanging out in the pool or wading in the ocean/lake/river/what-have-you, you'd probably be fine. But if you're going to swim or do anything beyond walking, I'd say find a way to secure it in place so it isn't going anywhere.
Last thing you want is to jump in the pool and see your dick floating by a few minutes later 😀