Rigorous exercise, cutting all the fat from your body, can reduce chest size... (NOT DIET! EXERCISE!)
But the chest may grow none the less. The tissue part can increase in mass even if your fat% hits rock bottom. It's not as likely, but it's not impossible.