I wonder how common this fear is, and whether it holds a significant number of people back?
It's certainly up there for me. Do I want to be depressed but reasonably attractive, or do I want to risk trashing myself physically (and there's no way of knowing you will until you've done it), and therefore end up with even more to be depressed about? You see some people, and just think, my god I could cry for you. My therapist is very honest about this as a factor, and made me laugh by saying she divides people into 'poor souls' and 'lucky bitches'. You can guess which side you'll come down on, but you don't know until you've done it, and it's not a fear to throw away lightly.
There is definitely an androgynous beauty that can work either way. There are also very good looking men, and very good looking women, who would never be attractive as the opposite. I suppose the thing to keep in mind is, the majority of the population, while generally not offensive (most people do have something), are not raving beauties to begin with, so if you're as hot as you perceive yourself currently, you may only be going down to average. But, if you are used to being related to as attractive, and got used to the doors that open for that, think carefully.
Maybe translate what works for you now into the future. Are you a classic dresser or are you alternative? Stay that way. Do you have a standard name or an outlandish name? Stay within those boundaries. Do you look good in some make up as a man? Get some more. What are your interests and emotions now? Be true to them instead of ditching them or changing (rather than broadening) them - the spark in the eye, that's what really makes someone attractive. You're 34, don't act like a 14 year old - that's barely attractive in a 14 year old. Don't fall into a stereotype. Less is always more. And lastly, some very hot people are deeply unattractive, because they're eating themselves up inside, and it shows.
I'm 34 this year as well. I understand completely where you're coming from.