Personally, I felt like it was more about *shape* than size - I could fill an A-cup before HRT, but it looked like saggy soft moobs without a bra on. The first time I could look in the mirror, with a tight shirt and no bra, and see the perky rounded cone shape with nipples thrusting forward... THAT looked like something I was uncomfortable going out in public with. I ended up wearing a super-compression sports bra under layers of shirts to hide them, in fact.
I do agree that it's also about proportion; when I was a rail-thin teenager, the tiniest bit of breast fat stood out a lot more than those A-cups on my larger, softer adult body. Someone who was clearly overweight with even C-cup breasts, assuming they were sagging and not highlighted by bra or outfit, could probably pass just fine for having moobs.