I've been doing it for like 2years now and never seen any real damage from it.. Just a little soap you know, mostly just regularly get it wet and let it dry and breathe.. Getting pulled and bent etc in your strides does it's own 'damage' like wear and tear and it can get sweaty and whatnot depending on your climate and activity level!
Just use water if you want but do it regularly.. Same for your harness or cock sock or whatever you use (if anything) - it's just about being clean bro, like any other body part, nothing fancy needed 🙂