I don't really think clothes are as much about painting "a fem side" as much as completing someone. At least for me in particular, I don't "transform" as much as show just another part of me that may still incorporate any number of stereotypically masculine and feminine things.
I was paying much more attention to women's clothes around the age of 11. Probably without coincidence, that's about when puberty kicked in, and it's probably natural that heterosexual men would start paying attention to things pertaining to women, especially since their clothes have traditionally been designed to attract the attention of men. For these clothes to start off as a "fetish" early on is pretty understandable in that context. So if you started getting excited about the clothes around this age, that's not surprising.
I think the idea of a man putting on "women's clothes" is sort of an interesting reaction, because what is probably the "intended" reaction is for a man to *look* at such clothes. However, there is an interesting morality at play. When a man's reaction to wanting to indulge in this aesthetic is to expect women to dress-up in this way, it diminishes women by reducing them to a role of serving the man's pleasure (not even necessarily consciously). If a man's reaction is to instead take on the aesthetic for himself, I think it shows much greater responsibility. Furthermore, if a man gains experience wearing these clothes and doesn't freak out so much when seeing them, he's more likely to keep a level head when he sees other people (mostly women) wear them -- and I think this is probably a good thing!