Well, I think it's useful to separate the things we might be dysphoric about into 3 general categories: anatomical sex, gender identity, and gender role/expression.
In my case, I have a problem with all 3, but if I had to pick the worst it would be the female gender identity. I've always, ever since I can remember, categorized myself with, competed with, been friends with, and compared myself against boys, not girls. Socially being a girl is wrong for me. Being a masculine girl in a female-dominated environment is just as wrong as being a feminine girl in a male-dominated environment because my issue is with the "girl" identity and not the gender role/performance that goes with it (which is why I'm one of a subgroup of trans guys who never identified as butches or socialized among lesbians, because I find no comfort whatsoever in masculinity when it's tied to femaleness).
But I also have issues with female anatomical sex - simply put, I find my "girl parts" revolting. I ought to have a penis. I ought to be bigger, stronger, faster, tougher, and a little bit hairier. And I think that alone would be sufficient reason to identify as male.
And I'm not really feminine. But that's really the least of my concerns, since it's perfectly ok to be masculine in either sex. If it weren't for the other stuff, I wouldn't care.