You know, I've tried to respond to this thread a couple of times since it was first posted, and every time I end up deleting it for one reason or another. For me, this is a touchy subject. The short answer is that yes, it increases the sense of wrongness I feel associated with my body image in relation to how I feel it should be.
Now to complicate things, the long answer is no. Most of the pants and shorts I own are "women's", same goes for socks and undies (in fact I don't own male underwear, TMI, I know). Essentially, most of my clothes are "women's". Obviously it's not about the clothes, it's the body they're on.