Gender roles are not inherently linked to a physical sex, but it do have a strong link with gender identities (bear with me a sec), the problem is that our society links gender roles with physical sex so that "boy's stuff" is taken as meaning "biological males natural activities".
Gender identities are a cultural concept, and their evolution was based on the presence of three elements, in this order of importance: gender role, supernatural sanction, gender presentation (not always). Please note that sexual orientation was not a factor at all, being attracted to someone didn't count as information regarding your gender.
Gender roles changes over time and place (think about weaving, it's a male or female activity? Ask an Iranian and a Native American and you'll get opposite answers), and in our time they are becoming more and more non-gendered.
Supernatural sanction, with the decline of organised religion, has been replaced by self-affirmation: you can see here a tenuous link with the time when natural inclinations where understood as a sign from the supernatural about the intended purpose of the child in the group.
Gender presentation has been the most variable, with specific outfits for third gender individuals and/or a fluidity in dressing manly or womanly or combining elements.
It's only when categories gets compartmentalised in a vertically structured society that this sort of rigid/causal mentality takes place: slaves and freemen, aristocrats and peasants, black and white people, man and women... two, distinct, by nature. It serves a hierarchical purpose, and so it needs clear bonds, no intermediate areas and no third options. And it need to be justified in an indisputable way: by nature/divine order. This is when causality kicks in: you are a noblemen because of your blood, you are a woman because of your sex. When there is competition for surplus resources, that being the reason for social stratification, there is an element of "us versus them" that makes stating your allegiance a priority and a fact set in stone by indisputable rules. We are still seeing that today.