The issue with anti-androgens is they don't necessarily suppress the adrenal glands' production of testosterone, nor do they halt the aromatization mechanism that converts T (however little of it still exists) into the more powerful di-hydro-testosterone (DHT), responsible for male pattern baldness, prostate enlargement, and a host of other ugly effects. This is why so many people on HRT take 5AR-inhibiting drugs like Proscar or Propecia (finasteride) or Avodart (dutasteride) in addition to their anti-androgens.
I'm hoping after my testicles are gone, things will get better. I'm not actually expecting it, but I'm hoping. It very much appears to be an individual thing, related to individual metabolism and biochemistry.