Breasts are mostly fat tissue, period. While there is a certain amount of breast tissue (determined by personally genetics), generally speaking the more muscle you get the smaller your breasts get. What is happening is likely one of the following things:
1) You may mistakenly be thinking your breasts are getting bigger if you're hitting your upper pecs harder because that pulls them up a bit more "perky".
2) You may be seeing the area as a whole start to get larger if you're hitting your lower pecs harder because the area is filling out more with muscle where it had been formerly flat.
The key in your case is to determine which heads you have been either neglecting or having difficulty hitting and target them while continuing to hit the ones that are growing very well as a secondary muscle group (pecs are secondary to triceps and visa versa). So if your upper pecs are growing larger than the rest of your pecs, focus on your lower, inner and outer with chest exercises and make sure to hit your upper when you do tricep exercises, for example. Avoiding hitting the head of a specific muscle group altogether isn't the answer and also make sure to change up your routine every few weeks as well. All professional athletes, including body builders change up their routine every few weeks and hit different muscle heads for each muscle group weather as a primary or secondary depending on their specific needs.
Another thing to think about.... picture professional female body builders Their breasts are almost non-existant unless they have had breast enlargement surgery done. In addition to my many years of being a professional athlete and the training required, I know many other professional athletes including body builders, both male and female and can say this with confidence.
