Biology is ultimately responsible for it. Not "men" as a group.
Strip everything down to a survival situation. A group of men, women and children with minimal provisions and resources. Someone must defend the group from threats. Someone must go out into the wilderness and kill food. Do you send the men out to do those things? Or do you send the women and the children to do dangerous and arduous tasks better suited to the bodies of the men? Do you think the women would volunteer to do that stuff and take their children with them to go hunt dangerous game, or face down an enemy tribe? Do women generally do this - offer to go out and do dangerous things or hard labor - or is it the men "keeping them down"? Fast forward to today when there is more ability than ever before for women to go into the military or do labor-intensive jobs if they choose and no-one is forcing them either way. What do they choose to do? on the whole, they do not choose to do these jobs, but prefer more comfortable, gentler work. Is this men's fault?
The infantilization of women is two-way. Men have nearly always been expected to put themselves between danger and a woman and her child, by women as much as by men. How many women would not expect this, that you know of? How many would expect a man to stand there and let her be menaced by a threat, or not carry heavy luggage for her, or not help her when she is heavily pregnant and not as agile as normal? Would such a man be considered "a good man" by women's standards? No. And such is the result of this biological "team", that this biology has informed our culture and cultural attitudes. It is not "men" doing this on purpose, it is men doing this because on the whole, keeping women safe from harm and doing the jobs that clearly put more of a burden on a woman than a man makes biological and logical sense. And if you have to be the one defending and working harder work, it has come to mean that men preside over matters of defense and physical labor, for the most part, and women preside and rule over the matters of children, and wherever they happen to spend most of their time doing that. (Show me a man that tells his wife or girlfriend how best to look after her own baby and is taken seriously by any social group. Show me a man who gets to "choose the wallpaper" instead of his wife, figuratively speaking).
Men are "in charge" traditionally because they were the ones way back in that tribal situation who were standing around without children hanging off them and whose job it was to co-ordinate safety, warfare, etc. But it's perfectly arguable women are as much "in charge" of any society by way of the next generation being in their hands, or by way of them choosing who even gets to reproduce with them. Women generally rule different aspects of society, but nobody rules every inch of it. Certainly not men. Men have become what they are largely because women require it. That includes being a species where we live in family units, where a father generally remains present to help take care of offspring, or is required to help pay for that offspring. Men - and culture - are also shaped by women's needs.
And why are men held to higher standards than women by other men? Whether you're talking about that tribal group, the modern military, the workplace, wherever... men have been the ones who had to get certain things done, and their survival depended on teamwork and performance. A soldier who didn't feel like fighting that day or was afraid might get an entire group killed. Men police other men not because it's necessarily some conscious, finely-calculated prejudice, but because it's probably in the DNA and instincts from thousands of years of this setup of men having to work in male-only teams, and having to bond based on shared values and motivations. That's why the gay guy gets picked on. That's why the "sissy" gets shunned. Because they are different and do not share the values and motivation of the others. It's not a nice thing, this nonacceptance of the different, but it is probably what powered the survival of our species in some ways. Biology is not a consciously nice thing, it's a cruel thing in many cases, but it is what it is.
So if you want to blame men for doing these things, you have to also blame biology, and you may also have to blame women for not wanting to do the exact same jobs and duties that men do. Because someone has to do them if they won't. I don't hear many women aspiring to be trash-collectors, sewage workers, day laborers, oil drillers, or cannon fodder. And I don't think that's because "the men are keeping them from it" any more. It's apparently because most of them don't seem to want to do these things. In countries with the greatest degree of freedom for a woman to choose her career, oddly enough, you don't find them lining up to do these things in greater numbers, despite vigorous encouragement programs to get women into new fields. You find them doing it in greater numbers in harsh environments where there is less choice and greater pressure to earn money to survive.
This topic is a lot more complex than "everything is men's fault because men rule everything". Men do not rule everything, at all. If they did, they generally wouldn't have to make themselves attractive to a woman's tastes or help a woman in any way in order to pass on their genes. They would only have to rape and walk away. And we are not a species that does that, for the vast part. Why do so many men on this forum and others ask about how to be an attractive man to a woman, and why do men everywhere bust their balls trying to be fit and successful and attractive? Well what's the point of that if we men just rule everything anyway? Just go and take what you want because it's a man's god-given privilege, and no negative consequences will possibly come of it. But wait, that's not how it works. . . you'd be locked up, and you certainly won't be an attractive long-term mate if you bring nothing to the table but that.
Women have far more power than many people can see or are willing to admit. Just because it isn't overt, or flashed in your face, doesn't mean it's not there. In fact, the power you don't see or notice is even more powerful than the one you do because it will not be recognized or challenged for what it is. Which is exactly what happens whenever there is a question about who is really in charge. Oh the men, of course. Bigger and stronger and always kings and presidents and bosses! Everything they do is only ever for themselves and they're a menace to everyone including themselves - for some... nebulous reason nobody wants to explain. So no man ever worked hard all his life to support his family, no man ever picked up a weapon to defend them or went to war for anything other than a thrill, no man ever dedicated himself to success so she'd put a ring on him, or got his teeth knocked out defending her honor. Everything he does is nefarious in nature. Man is, through and through, an entirely self-serving and self-harming creature that when harm befalls him, is always in some way deserved. Because he apparently planned the roles and responsibilities that nature doled out to him, and took the shape that was demanded to survive. Clearly he planned culture, rather than it springing from the circumstance in which humans found themselves, and he gets to dictate exactly how it flows, at each moment.
No. Not everything is men's fault, or men's design, or men's conspiracy. If it was, don't you think we'd have improved our own lot, by now?