aaronst,
I fought with that line of thinking for years. Out of my own ignorance, I secretly thought that since I wore women's clothes that I had to be gay, but I knew that I was not gay. This left me with a big hole for many years. I was of the belief that cross-dressers were transvestites and drag-queens and that transvestites and drag-queens were some form of gay men. (I now know better, I don't mean to offend anyone, it was just my ignorance at the time.) So, since I knew I wasn't gay, that meant I wasn't a transvestite or a drag-queen but it didn't explain why I felt the need to wear women's clothes. It also never explained why I always felt that I wanted to be a girl, or that I should have been a girl. Well, I finally found my answer a little over a year ago. I'm transsexual.
I'm not gay (well maybe lesbian). I'm transsexual, and I'm okay with that.
I'm also not saying that since you like to cross-dress you must be transsexual. Not at all. There are a lot of reasons men wear women's clothes - it could just be as simple as you love the feel of the clothes. As Janet suggested, if it is something that is concerning you (and I'm assuming coming here and asking, it concerns you) then maybe seeking a therapist to help you sort out these feelings, urges, whatevers.
And vexing, there is nothing wrong with being gay, if you are gay. There is a problem with being homophobic. I'm not homophobic. Jury is still out on the gay part. Not sexually attracted at all to men (sorry guys). At least not at this point.
Deanna