It's not about "weight".
Yes, you can reduce the size of your chest by lowering your body fat percentage, but you only get there by increasing your weight. In other words, by doing athletic things to increase your muscle mass and strength.
See, if you try to diet them away your body will go into "starvation mode" and cling onto every scrap of body-fat it has.
It "is" possible that the non-fat-tissue is expanding/growing, in which case, nothing you can do to reduce their size, least of "loose weight"