To me it does not make much sense indeed. Actually, I'd think it's much better to even start with just anti-androgens, and then start E when your T is dead. Then hormonally, your development is just like a girl's, and that sounds better to me.
But it's not like your doctor is a mad scientist either. His approach does make sense in a way. Basically, in some people, just putting in E is enough to bring the testosterone down enough to make E effective. And as a general principle in medicine, you should give the smallest amount of medications required to achieve the desired results. Plus, it is true (especially in older patients) that anti-androgens are a pretty violent thing. Spiro empties you of your water and sodium, while cyproterone literally poisons testicles and is hard on the liver/kidneys/whatever.
But I disagree with that, to a point, when it comes to HRT. After all, the desired results are the maximum results, and how do you know whether you're having the maximum results if you don't ever try anything higher?
Besides, to both suppress T and get results, the dose of E required is generally higher. So there we're in a debate: What's worse? A ton of E or E and anti-androgens?
I think you should insist a little for getting anti-androgens if you believe they're right for you. After all, if testosterone is well under control, after a while, out of safety, your doctor and you can always try to slowly bring the dose down and see how your levels fare.