Well its just his lack of information and this also can help fuel the anxiety.
Do not listen to anyone other then your doctor, they are the ones who know you and what is best.
Before i went on HRT i was on major strong anti-depressents and anxiety meds. Once i started hrt, went full time and started my RLT i ended up stopping everything other then HRT. Its funny how once you get over your minds fears how much the others melt away.
Good luck and as i said you are you, no one know your other then you. Be honest and trust in your doctors and you will be ok.
Izzy