It shouldn't do you any harm. I had to quit HRT for 3 weeks for GRS, and nothing significant changed; in my personal experience, by week 2 T rebounded (I got that male smell back and That Thing reinflated to its original dimensions) but there wasn't time for it to do harm in terms of permanent changes. Breasts, skin, fat distribution, and facial changes remained the same.