When my mom finally took me to therapy after getting caught for years, his advice was 'so buy him a dress' - never did thank that guy enough.
When I go out, I try to dress as a woman of my age, weight and income would dress. Stylish and not exaggerated, nothing that screams out at anyone. I don't know if I "pass" or not and I really I don't even care and it not my intention or goal. I just out being me that's all and if someone has a problem with it, it is really their problem, not mine. So long as I feel gracious and act dignified, I'm cool with myself. I try to take myself seriously, (maybe seriously isn't quite the right word, because I don't mean it in the sense of grave or important - I'm thinking of credible, plausible and really, something more along the lines of legitimate) and find that in order to be taken reasonably I need to first present myself as a reasonable person. In order to lessen the public perception of dressing as some sort of perversion it is necessary to go out in public wearing suitable styles and maintaining a dignified bearing ( i.e. not dressed or acting like a pervert) with gracious manners and pleasant mannerisms. Extolling an attitude of Please and I beg your pardon and even, if necessary, By your leave, and less one of IN YOUR FACE. People often agree to things when nicely asked, that they strongly object to when they are just rudely shoved in their face. Too many people think that manners are only a means of having to beg and plead, and that is not the point, at their best, manners are a method of subtle corrosion that operate under the guise of a social lubricant. People liked to be asked, or even better, to think that its their idea in the first place
For the most part you are free to do it as you wish to be - once you consider the situation and the circumstances. If you want to dress up and go out to a modern techno rock concert in a trendy nightclub you will probably be OK. But think twice about dressing up for Homecoming in College Station - Texas A&M vs. Oklahoma State. (Actually think twice about that little slice of life, even dressed as a male, hell – even dressed as a cowboy.)
Last time I went out in New York all dressed up I walked about 10 blocks, and those are huge blocks, including having to traverse the entire length of Times Square, the only people who seemed to notice at all were obviously tourists. The natives all have that uniquely big city attitude of ... Big deal, your the fifth guy in a dress I've seen ... In the last half hour! San Francisco in the 1980's was even more blasé, if that is possible. On more than one occasion in SF I was almost assaulted by people who wanted me to know how cool they thought I was for dressing up like that, going out in public and Challenging the stereotypical roles that society has arbitrarily created for men and women.... blah, blah, blah .... on and on for forty five minutes, you know the type. And that's not what I thought I was doing anyway. I just wanted to feel pretty, grab a beer and dance a little bit. I doubt that New York is any different now than SF is, in that no one really cares much at all.
The worst treatment I get as a general rule is dirty little looks from uptight Christian Right types and I really don't give much thought to what they think anyway. Yeah, I've got a few catcalls, hoots and hollers over the years, but that's no big deal. I've never got into a fight, or arrested or anything bad. So I guess that the worst part of all this is what? Potential embarrassment? A few people don't much care for it, and might even call you on it? OK, so what? They probably voted a straight Republican ticket and I'm not too wild about that - heck, I'm not all that thrilled about Democrats either these days.
As long as you do no harm, you should free to feel about it as you wish. But that does not necessarily imply that everyone is going to get up and cheer for you or start a national holiday to celebrate the wonder that is you. To expect society to change (which it is always doing anyway) on this point is shallow, superficial and particularly self-absorbed. Besides, what is this "Society" thing anyway? No "society" ever showed up on my door to tell me a damn thing. For the most part, I think "society" is really the voice of your parents and yourself. Clear that up and you will find, as you have, that "society" doesn't have as much of a problem with this as you think.