I agree, individual results vary greatly. I am 17 months on T and have seen no increase in my sex drive at all (I think I must be one of the odd ones who don't), my aggression (though never much to begin with) is less than before. I never was competitive and still am not. I am pretty much the same person I always was, just happier and more at peace.
I had PCOS and endometriosis before starting T and it certainly didn't make it worse. I had a hysterectomy/oophorectomy a few months ago because of continued pelvic pain even without periods and was happy to see those parts leave.
As stated above, you may get acne, you may see higher risk of heart disease, etc, but every person is different and depending on your family history, you may or may not have those issues anyway.
Bottom line is that you need to make the decision for you, not based on anyone else's opinion of what is right or wrong.