again according to the biology lesson, one function of testosterone is the stimulation and regulation of growth, development and maintenance of male reproductive organs and secondary sexual characteristics. so i guess, without the right amount of T the organs do not get proper care and shrink maybe, or maybe estrogen has some effect on it as well.
edit: disabling of course, T is required for the regulation and stimulation of sperm production, so lack of T is quite probably the reason