"To be part of a privileged group" - this really needs to be cleared up. Either we are saying that both males and females are privileged in different ways but BOTH have a degree of privilege over the other, or we are dealing with the "men just have power over women and women have no power over men and never had" spiel.
It is a fact that women have power over men too. It is usually SOFT POWER, rather than "hard power" and a little more covert but it is power. If you don't know the difference between soft and hard power go look it up. It is real. Men might be bigger on the whole than women but women can and do use men to do their fighting for them, for example, and that is also power a person may wield.
If woman A and her boyfriend are in a bar and man B walks up, hits on her while her boyfriend's at the bar and gets rejected, and then follows this up by pushing her to the ground, that's hard power - the woman isn't likely to pose much of a physical threat to him. However, if woman A then goes to her boyfriend, tells all and suggests he knock man B's teeth out, and he does, that is soft power. She isn't as strong as man B but it doesn't matter. She used her position and her influence to get the job done. In fact, I would argue a person who can get others to do their dirty work for them without lifting a finger have a lot MORE power than those who have to do their own. They are less likely to be injured or blamed for starters. Woman A might even have her boyfriend's pals pitch in too. Soft power can offer a lot of interesting options.
I know damn well women exert soft power, and women are perfectly aware of it - whether it's getting men to defend them, help them get something off the top shelf they can't reach, to utilizing the male tendency to want to protect and take care of a female who is close to them. Men are absolute suckers for it, and women know it. And they have to be, because biologically women always needed men's help in the natural world when we were building tribes and civilizations. Women needed men's utility. There is absolutely no getting away from this - the human female is unfortunately vulnerable while carrying children and survives much better when women help women and men help women too. We evolved that way.
Women are very good at getting men to do what they want. I'm sure they evolved that way too, because pregnancy always foisted some degree of dependency on a human female - she needed to be influential to survive. That soft power is less visible than a "big strong man" but it is there and anyone downplaying it likely either has an agenda or is in some sort of denial, choosing deliberately to focus on the most visually prominent rather than the most potent.
And as for this idea "male privilege" "exists for all men - but actually no, only for some" is ridiculous. Either male privilege is by definition a privilege one has by virtue of being male or it isn't. You don't get to have your cake and eat it too.
Yes, sexual dimorphism sucks if you think being made to make use of soft power rather than hard power sucks (actually small men are typically made to make strategic use of it over hard power, they can't always compete with a 6'4" opponent - but not being women they don't have all the various strategies and lures available only to females at their disposal), but it is absolutely not the case that men have all the power or influence in the world at all. If you think that then you are purposefully denying the power of women, and the general willingness of men as a gender to defend them, provide for them, please them, and avoid treating them with the severity they treat other men. Women not only have the advantage that men tend to like women more than other men, whom they are more likely to see as potential rivals, but that women tend to side with women too. Just about the whole of society including men cares more about the safety, wellbeing and provisioning of women. Human society itself is woman-centric at the core, not man-centric - the basis of the family is the mother and child and men are the "helpers" of it. I don't know how many times I have to lay out the obvious, that which is there for everyone to see if they get their heads out of current dogma, and stick them out of the front door and have a look at the real world.