Even with frequent and consistent cleaning soft packers are going to show wear and will need replacement. You can pick off the lint, just be careful not to tear the packy itself.
I don't use basic hand soap (or stuff with perfume or cologne scent in it.) I use bar of olive oil soap, it works pretty good for keeping soft packers clean.