Hello Trish!
These feelings are completely normal. When I first came out of the closet it wasn't long before I realized I was more than just a cross-dresser, I was truly transgender. BTW... I had just turned 40!
I was only slightly interested in HRT, but really wanted to approach it on my speed. I began dosing myself with herbal estrogen (and yes, despite what some people might say, it really does work - at least it did for me).
My breasts began to grow and I started worrying about what my doctor would say or people noticing. I kept on thinking "I can stop anytime, just a little bit more". The truth is, I couldn't stop. I was happy with how my body was changing.
That winter (about 8 months out of the closet and 6 months on herbal E) I had a serious psychological struggle over my body and my future. It was like my male self was trying to hang onto who I used to be, while my female self was finally being born and taking over. I was under the care of two different therapists, BTW.
It didn't take too long for me to realize, no matter how much I told myself "I can abort this at any time"... it was just a safety net. I knew deep down inside, there was no other path than forward, no going back. I had "purged" so many times in my life with the thought that I was done with it, only to find myself back in the closet a month later.
Once I was able to embrace this notion, everything got easier. I dropped my old primary and got hooked up with an LGBT friendly medical center and started planning my transition. My new doctor persuaded me to ditch the herbals in favor of monitored pharmaceutical HRT. I've been medically transitioning for 10 months now and my only regret is that I waited so long in life to start.
Now I'm out to my close friends, my mom and my daughter. Honestly, every time I tell someone, it brings more relief and I feel better about myself.
My next challenge is coming out at work. So far I've been able to hide my changes as it's been somewhat slow, but at some point (very soon) I really want to throw all my "men's" clothes away and be done with all of this secrecy.
Given my experience, I'd say you really don't know until you try. There will be some physiological and emotional challenges. But once you can start being honest with yourself and the people around you, you'll feel free to live your life how you want to... and that's something I had been waiting for, for 40 years!