Do you have a gender therapist? Having a letter from a therapist helps the Doctor help you. HTR has significant effects such as becoming sterile and growing boobs; put yourself in the Doctors shoes. How does he know your need for HRT? Is he/she in jeopardy of a lawsuit if he prescribes on your word alone?
I am not saying do not confide in your Doctor, you should, but he/she may request a letter from your gender therapist.