Most anxiety is fear based, and fear of the unknown sounds plausible in your situation. Fear of what higher doses mean to your future, fear the doctor may not know what he is doing, fear the sky is blue.
Now step back and think, is it the doctor visit? Probably not, what are your feelings on transitioning? Will it adversely affect your family and social life? Are you concerned what you will look like after HRT? There can be alot of anxiety surrounding these things.
If you are comfortable, please share some more information about your life surrounding HRT. It could be enlightening and help us figure this out