Shawn, you need to consider that breast are comprised of mainly two types of tissue.
The adipose, or "fatty," gives breasts their shape and mass (marked as "7" in the diagram from Wikipedia).
Glandular tissue, which includes the lobules and ducts, act to facilitate the production of milk (3 and 6).

Male-bodied persons with "pseudo-gynecomastia (i.e. "moobs") are usually overweight, and their breast tissue is almost entirely fatty tissue.
Males with true gynecomastia. like myself, have both glandular and adipose tissue.
Some people have both conditions.