I understand your point.
But I suggest religion isn't so much at the core as being used. The core are those who seek to enforce their notions by any means.
That may be religion and frequently is. When the texts don't say what is required, they are fabricated. This was done by the Popes and others for almost 1000 years, until finally, people began to read the texts for themselves.
Other justifications may be science. Eugenics. Claims that children from India carry diseases. (Used by some idiots here in the UK, in the 60s and 70s).
Science can be preverted as much as religion, perhaps more so, since few understand the science. It can't be checked, since experts can always over rule and there is always another expert. Climate change for example. In the late 60s we were told there would be standing room only on the planet by 1999. In 1990 we were told sea levels would rise by a metre by 2012. (I live on the coast). I personally find psychology so utterly laughable and can't understand how anyone can take it seriously. Yet few dare question psychologists.
But the answer is quite simple really, it's politics. Absolute liberty.
The notion that we each have a basic right to live as we choose. As basic as life itself. Restrictions can only be justified to protect the liberty of others.
I don't think the world is yet ready for absolute liberty. Most would seek to forbid, those choosing to, to walk around naked for example, to take an extreme case.
Religion and science are, in reality, two sides of the same coin.