Annah, as I said, both factors count. Make most above-average-breasted women lose a lot of weight, and their breasts will shrink. How much they shrink will depend on the person, but still. It's by no means always true, but most skinny girls are going to be in the A realm, just like alyx, and I doubt, unless they just can't gain weight, they're all doomed to that size. So I don't think it's accurate to declare that having small breasts means you weren't receptive to HRT and thus you won't ever get much of them.
I say, as long as some breast tissue has developed (that is, having something there as compared to no breasts at all), it is possible for fat to deposit there in a female pattern. And just like some women have more fat on the hips or thighs or butt or whatever, I think that there's a different factor that influences how much fat goes in the breasts.
In a mathematical way, we could say that in the equation Size = Ax + B, x being weight, everyone has both a different A and B. Or so I think, at least.