The way I have always envisioned an answer to this particular question (or what makes a man a man) is sort of drawn out, but makes sense in the end. If someone says "you have breasts, you are a woman." Then what happens when that person has a mastectomy? Is she no longer a woman? If they continue with "alright, but you have ovaries and a uterus." Said woman has also had a hysterectomy. Is she no longer a woman? Now the clincher comes down to is it chromosomal or is it hormonal? Chromosomes do go awry which causes abnormalities. Hormones not only affect the body physically, but also the brain\mindset of a person. If a male has the so-called "man boobs", does that make him a woman?
To make it brief, the above answer is a much more simple and exact answer. However, if you really think above the above scenarios, that is something which could be used with someone should the topic arise in conversation.