I took only antiandrogens for a while prior to adding estrogens and stuff.
Go and look at Avodart forums and discussions. Sure you can take antiandrogens, men do it all the time for their hair. I read about some who make a paste out of spiro and rub it on their bald spots, others who break open their avodart gelcaps and all other sorts of things. The side effects they complain about are the yumyums to us. You do need either testosterone or estradiol, yes, but you don't need as
much testosterone as your body is making and a doctor can help you reduce the levels without needing to supplement with anything else. It's no fun to bottom out though and not have enough sex hormone of either flavor, so make sure your doc isn't a goof.
Seriously though you'd be missing out, estradiol is the best part

Prolly the best thing to do would be to sort the issues barring you from hrt out first so they don't get worse.