A lot of it has to do with how you gain muscle. In human biology class we learned of three main types. these types determine your shape and muscle gain/loss. These body types are noticeable even through our Dimorphism.
Ectomorphs are the humans with smaller frames, Muscle is harder to gain and harder to lose high metabolism.
Endomorphs are larger framed people, heavy set individuals with a lot of body fat that hides muscle.
Mesomorphs have low body fat, gain muscle easily but can lose it quickly, larger frame (broad shoulders, stocky rib cage etc)
I am an Ectomorph, you and I share very similar height and weight, but for me it took about a year for me to lose any Physical strength but when it kicked in, I lost it fast, and my muscle mass has barely changed, however fat has been redistributed.