Very crudely put.....
There are two answers. Firstly most men do no care about clothes. As long as they are warm or dry or protective that is good enough.
And then there is the more sinister reason
The most dangerous males are the repressed homophobes and transphobes
A woman dressing as a man will be ignored by these dangerous types because they will never admit to a sexual interest in someone else who appears male.
A man dressed as a woman (or even a transwoman) will be seen by homophobes as female until he finds out and will then be regarded as a male. To the homophobe this "outs" him. Everyone will know the truth, his friends will laugh, his family will mock him. So he "proves" his "manhood" by reacting violently. By beating that person up or through violent assault he shows he was tricked and how angry because is a "real man". These sorts will never agree to anything that might allow another man to look feminine because they might make a "mistake" or worse still be attracted.
In truth it is he who needs help. Homophobes often suppress their inner natures just as many trans people do.
If they got help we would ALL be safer and happier