I started HRT in April of 2014. I have a pretty active job, so I'm not able to let my mass go down as much as I'd like to.

That said, does it matter? I'd like it to go down still, but is it gonna change on its own? Is it gonna be better if I stop doing so much physical stuff?

I think you'll find it will change regardless. Women can be pretty darn strong with or without bulging muscles.