The longer you are on HRT the longer it will take for the body to return to previous levels, and even then the physical changes won't be noticeable for some time after. A few weeks off for surgery, especially since you've been on HRT for three years should not be an issue. After three and a half years I recently went off HRT for a few weeks prior to GRS. Other than some headaches (which may have been related or not) I can't say I noticed any difference.