From what I've heard from different surgeons and doctors is that exogenous testosterone does increase the chances of a blood clot in general. Since surgery also increases chances of blood clots, some surgeons prefer patients stop hormones in advance of surgery. Anyone taking hormones not produced by their body (including cis-gendered males on hormone replacement therapy) is at an increased risk of clotting. Every surgeon is different and, as mentioned, has to be concerned about potential risks, even if it is only a significant risk for a small population (for example smokers or people with genetic clotting disorders, or the rare otherwise "healthy" patient that ends up having an issue).
With that being said, my top surgeon did not require me to reduce or stop hormones for any period of time and my hysterectomy doc only asked that I delay my injection after surgery by a few days.
I think it's worth a discussion with your doc based on your age and lifestyle if it is truly something you need to do but I don't know that I would ever recommend not following your doc's advice, even if the chances of it being an issue are slim. Perhaps he'd be willing to negotiate some kind of middle ground you're both comfortable with?