A lot of us have acquired clothes, purged them, and repeated, many times. The fact that the urge never went away convinced me that there must be something real, a basic part of who I am, behind it. And the chance of getting busted convinced me that I had to get it out in the open before I was outed against my will.
You are right, it never goes away. It is not something you can ever forget. Good luck with whatever path you choose.