I had most of the same symptoms you describe when I was your age (except my breasts never grew very large and I've had two children). You could be XXY, or, as Serenation says, maybe your mother was given hormones while pregnant with you (which can mess up your endocrine system and prevent your brain from masculinizing properly).
Whatever's causing it, from what you describe, you certainly have signs suggesting your body is producing below normal male levels of testosterone. One way of narrowing down what might be causing it is to have your blood hormone levels tested. The ones to look at are: total testosterone (and free T if it's available); SHBG; LH and FSH.
Aged 24, your total T should be near the top of the lab reference range, somewhere between 700 and 1100 ng/dl. If it's significantly lower than that, it means that the symptoms you've described are probably due to low testosterone.
The other measurement to look at in conjunction with total T is SHBG (sex hormone binding globulin); if that's high, it binds up nearly all the testosterone in your body in an inactive form, in which case even if your total T is normal, you can still develop symptoms of low T. For someone your age, the optimum for males is around 15 - 25 nmol/l, if it's significantly higher than that, then it's likely to be a contributing factor in your symptoms of low T.
If you have low total T and your LH and FSH are near or above the top of the range, you have primary hypogonadism, and you're probably XXY. Low total T and LH/FSH that are towards the lower end of the lab reference range indicates secondary hypogonadism, and mean that, with fertility treatment, you'd have a good chance of producing viable sperm (in which case, if you want to have children in the future, I'd suggest banking some before you go on hormones).

This is a blood test I had done last year. In it you can see how my testosterone level is kind of low (15.2 nmol/l is 438 ng/dl), and my SHBG kind of high. While neither is outside the lab reference ranges, the combination of the two results in me experiencing symptoms of hypogonadism. If you look at my LH and FSH, they're both low (and the FSH has been flagged as out of range), indicating that I have secondary hypogonadism. If I had primary hypogonadism, they'd both be near or above 10 IU/L.
In my case, based on other symptoms I have, it's probably the result of being exposed to artificial female hormones during my prenatal development.