From what I've read, the consensus in the medical community today is that sex hormones are generally well-tolerated. There are negative side effects in some people, but they are generally rare and well managed with regular annual physical check ups.
I tend to give greater weight to the positive side effects, which for me, bigger boobs, fuller hips, softer skin, feeling right about myself, far outweigh any negatives.