If you are injecting, it's likely a reaction to the oil it's suspended in, try another brand. If you are taking pills, again, it's probably one of the fillers or additives that you are reacting to, tray another brand. Patches, it's almost certainly the adhesive.
Now, despite the reactions of some around here, allergy to estradiol is possible, but it is incredibly rare. I can't find a case study. There is/was some data to suggest that estradiol played a part in anaphylaxis, but this was proven not to be the case. An allergy to estradiol would likely have revealed itself before HRT, as some is present in all humans.
EDIT: Are you injecting and do you have a peanut allergy?