It took 2-3 years for my upper-body musculature to clear out - not that I had much by any stretch of the imagination, but it was still a tough rather than a soft physique.
I knew nothing about nutrition and ended up on a calorie-negative diet pattern for awhile, which definitely helped soften my upper body, even while hauling equipment each day that was far too heavy for me.
I can still muster up the strength to move a desk or other large object when I need to (genetics: both parents are fiendishly strong) but these days a couple bags of groceries is plenty for me, and I don't have as much stamina as before.