Hi Sarah,
There are a lot of us that "suffer" from one condition or another that adds to the confusion we face. I am of a similar age to you and similar family status.
On the transition front, you ultimately have to make your own decisions based on the balance of your life and your heart.
...BUT with regards to your medical condition you need to discuss this with your doctor in confidence and get a better understanding of whether there are any other reasons for the T prescription.
The other common side effects are listed as:
Loss of libido - reduced interest in sexual activity
Impotence and loss of spontaneous erections
Tiredness
Reduced muscle power and stamina
Change in body shape - increased fat and breast development
I guess this is where the comments from your wife may be coming from, as opposed to the lack of body hair.
So let's say you can live with these, my real concern would be that you risk the following more serious effects if you do not take the T :-
Osteoporosis - brittle bones resulting in fractures
Early heart disease
Depression
The first two are potentially serious now if you have any previous history, and they will become more serious in the next decade for you. The latter is likely to lead to other serious issues in your life, and will not help your disphoria.
So as someone with a family myself, I would advise that if you have no intention of transitioning, then seriously consider taking the T, but get a better perspective on any other reasons for the prescription, and see if you can keep the dose level down ( ie suggest concerns over getting too aggressive ).

Chrissty