That's an... excuse the pun, old wives tale.
Breast growth is geneticly linked, If you were to start hrt at puberty, you would roughly develop the same size breasts as your female relatives. However, the later you start, and the lower the levels of human growth hormones present in your body, the less likely the puberty rate of growth will be. You may get swelling, and a-b, but its unlikely to go higher naturally, after 30 ( I say unlikely, not not)
The younger one starts hrt, the more problem breast growth and visible change will be.