History as a Giant Data Set

[lpetrich]

History as a giant data set: how analysing the past could help save the future | Technology | The Guardian
Peter Turchin Home - Peter Turchin - a leading advocate of that kind of research.

Nature magazine's first 2010 issue described things that we could look forward to in 2020, including brain-computer interfaces, fast-growing crop plants, and fossil-fuel dependency soon ending. But a few weeks later, the magazine published a letter that warned of trouble - social trouble. Societies tend to go through rising and falling phases, and social indicators in the industrialized world suggest a falling phase, complete with a strong possibility of major unrest in our future. The author of that letter was biologist turned historian Peter Turchin.

He had cut his teeth on periodical cicadas, insects that evade their predators by emerging in such numbers that eaters of them quickly become stuffed and unable to eat many more of them. Their cycle lengths are 13 and 17 years, prime numbers, because they get out of sync with predators' cycles, which are often shorter.

Turchin’s approach to history, which uses software to find patterns in massive amounts of historical data, has only become possible recently, thanks to the growth in cheap computing power and the development of large historical datasets. This “big data” approach is now becoming increasingly popular in historical disciplines.

Fragile States Index | The Fund for Peace - the US and UK are becoming more fragile, while many other nations are improving.

“We are in an age of considerable turbulence, matched only by the great age of Atlantic revolutions,” says George Lawson, who studies political conflict at the London School of Economics, referring to the period from the 1770s to the 1870s, when violent uprisings overthrew monarchies from France to the New World.

Turchin sees his prediction for 2020 not just as a test of one controversial theory. It could also be a taste of things to come: a world in which scholars generate the equivalent of extreme weather warnings for the social and political conditions of the future – along with advice on how to survive them.

Turchin had come out of research into ecological cycles like the cicada ones.

Lemmings do not commit mass suicide, as Walt Disney would have had us believe, but they do go through predictable four-year boom-and-bust cycles driven by their interactions with predators, and possibly also with their own food supply.

They breed until their burrows are overcrowded, then they depart from their burrows and seek new places to live. Many of them die as a result, and their population crashes to near extinction.

Many historians are very skeptical, however. How does one quantify history? How can we be sure that some proposed cycle is little more than pareidolia? Like features in clouds.

When Turchin got into this research in the late 1990's, he discovered that he was preceded by 2 decades by Jack Goldstone.

At the time Goldstone began his research, in the mid-70s, the prevailing view of revolution was best understood as a form of class conflict. But Goldstone made two observations that did not fit that view. First, individuals from the same classes, or even the same families, often ended up fighting on opposite sides. And second, revolutions had clustered in certain periods of history – the 14th and 17th centuries, the late 18th-to-early 19th centuries – but there was no obvious reason why class tensions should have boiled over in those periods and not in others. He suspected there were deeper forces at work, and he wanted to know what they were.

 

[steve_bank]

The old saying is 'those who ignore the past are doomed to repeat it'.

Conservatives as a bad ass military power. China in ways is acting like pre war Imperial Japan. Like the old Soviets Russia is out to destabilize the world.

Technology and environment is not the biggest threat. IMO.

 

[lpetrich]

Then he worked as a teaching assistant for Harvard demographer George Masnick. He worked on describing the consequences of the baby-boom population bulge, like the social upheavals in the Sixties era, roughly 1965 - 1975. This gave Goldstone a clue, and he looked for premodern population data. He found some, and he noted “It was astounding: there really was a three-generation surge in population growth before every major revolution or rebellion in history.”

He came up with a Malthusian sort of model, where both commoners and elites multiply until they surpass their resources' carrying capacity. Commoners outbreed the conveniently arable land, and their standard of living falls. Elites grow until they greatly outnumber the higher-level and more prestigious positions that they might possibly get, and they start fighting among themselves for those positions. The leaders might try to pacify the commoners by capping rents and the like, but that displeases the elites even more. In the fighting, some elites may seek allies among the commoners.

Goldstone suggested ways of measuring mass mobilisation potential, elite competition and state solvency, and defined something he called the political stress indicator (psi or Ψ), which was the product of all three. He showed that Ψ spiked prior to the French Revolution, the English civil war and two other major 17th-century conflicts – the Ottoman crisis in Asia Minor, and the Ming-Qing transition in China.

But what triggered a revolution was some chance event that pushed society past its breaking point, and what happened next was much more difficult to predict.

As he was writing his book, the Soviet bloc was unraveling, and he noted its increasing psi value in then-recent years. Much of the developing world also had high psi, and the US's psi is headed in that direction.

Peter Turchin had personally witnessed a society that had done a lot of self-destruction.

He was born in Russia, but his family defected to the US in 1978, and he did not return to Moscow until 1992. “That was the year things collapsed completely,” he recalls. It was December – “dark, horrible. There were drunk people lying everywhere.” He and his wife passed a blown-up car on the way to the market and watched mafiosi extort cash from terrified stallholders while the police looked on. These were images that stayed with him.

Turchin decided to go further, back some 8000 years, well into the European Neolithic. Greenland ice cores trap pollution from economic activity, aristocrats' villas reflect elite competition, coin hoards reveal anxieties about strife, and skeletal malformations indicate malnutrition. Very patchy, but enough to show us trends over time.

He found not only Goldstone's cycles over some 3 to 4 centuries, but also shorter "fathers and sons" two-generation cycles, roughly 50 years long. One generation rebels, then its succeessor generation is unwilling to go through that rebellion's strife. But that successor's successor has less memory and more injustices, and then rebels.

 

[lpetrich]

Peter Turchin and his colleagues have created Seshat: Global History Databank - named after an Egyptian patron goddess of record-keeping. It contains data on the societies of 30 regions scattered over the globe, with its contributors doing that to avoid the labor of doing everywhere in the world. They divided the world into 10 major regions, and then selected 3 natural geographic areas in each region, one with an early-developing complex polity, one with an intermediate-developing one, and one with a late-developing one or none before the early-modern colonial period.

Seshat contains coding of the polities with a large number of variables: social complexity, warfare, ritual, institutional variables, legal system, equity, social mobility, religion/ideology, well-being, economy, agriculture, and population. It's very detailed, and the scorings were prepared with the help of specialists on the various societies.

Back to that Guardian article.

Turchin decided to extend his work to industralized societies, and a good one is the United States. It is (1) one of the longest-industrialized nations, (2) it is relatively unaffected by other nations, and (3) it has been politically unified. The third criterion rules out Europe as a whole, the second one individual European nations and Canada, and the first one most other nations.

He chose variables more suited for an industrialized society. "These included real wages for the mobilisation potential of the masses; filibustering rates in the Senate and the cost of tuition at Yale for elite competition; and interest rates for state solvency." Psi was low in the Era of Good Feelings, indicated that that era was very aptly named, high in the 1860's, around the US Civil War, and low in the years after WWII. Since 1970, it has risen steadily.


Turchin has considered how to avoid further troubles. He once addressed an ecological problem: attack of pine trees by pine beetles. Pesticides had a problem: they also killed a beetle that preys on the pine beetles. He proposed an alternative: cutting down and removing the affected trees.

He is now considering human societies and how they respond to high psi, high social stress. They do everything from rapid recovery to social collapse. Britain recovered very well from the 1688 Glorious Revolution, but the Soviet Union had the Chernobyl accident as a contributing factor to its breakup. Some polities disappear outright, it must be noted, like the Western Roman and Eastern Roman (Byzantine) Empires.

 

[lpetrich]

To understand this phase of the cycle better, Turchin and others plan to build computer-simulated societies composed of thousands or millions of individuals – so-called agent-based models – and programme them to behave according to the laws they have deduced from real societies. They can subject those simulated societies to stress, for example by injecting a virtual youth bulge, and observe the downstream effects on state, elites and masses. Once Ψ has reached dangerously high levels, they can add a shock – in the form of a foreign invasion, say – or increase resilience by strengthening the society’s infrastructure, and see how it responds. They can ask such questions as: what does it take to nudge a society in crisis toward total collapse? What interventions would divert it towards a less bloody outcome? Why are some societies more resilient than others?

Societies are much better at reconstructing after disaster than at preventing them, though there are exceptions, like the FDR's New Deal. The US elites agreed to share their wealth more equitably in exchange for the common people leaving them in power. Though there was the Business Plot by some business leaders against FDR, a plot that did not get very far.

Goldstone has continued to present his ideas.

At a workshop on societal collapse at Princeton University last April, someone asked him why historical societies had so often failed to act even when the signs of a looming crisis were impossible to ignore. He suggested it was because elites continue to live the high life for some time after things start falling apart, buffered from the upheaval by their wealth and privileges.

That may be why climate troubles have had so little action for alleviating them - until recently, it was not very obvious that there is much to be concerned about. But heatwaves in Europe, droughts and wildfires in California and Australia, and big hurricanes are now very evident.

There was a little bit of action taken: development of renewable-energy sources. That development has borne fruit, with wind turbines and photovoltaic cells now competitive with fossil-fuel electricity generation. So it gives a hint of what is to be done.

 

[lpetrich]

Turchin believes that historians will soon embrace complexity science, just as biologists did half a century ago. They will come to realise that it allows them to see deeper and further, to discern patterns that are not visible to the human eye. In fact, it is already happening.

Like the Center for Existential Risk at the University of Cambridge in the UK. It studies biological, environmental, and technological risks, including artificial-intelligence risks.

To Turchin, these are all encouraging developments, but 2020 is nearly upon us, and lawmaking institutions in both the US and the UK are now so divided along ideological lines that they can barely function. In both countries, disgruntled members of the elite have taken power in the name of the people, while failing to address the underlying causes of the malaise: widening inequality, a swollen elite, a fragile state.

Donald Trump and Boris Johnson.

 

[Politesse]

God save us from biologists wandering into the social sciences and wondering why we don't just run some simple models. Always with briefcases full of oversimplified analogies and just-so stories.

 

[lpetrich]

God save us from biologists wandering into the social sciences and wondering why we don't just run some simple models. Always with briefcases full of oversimplified analogies and just-so stories.

This isn't some handwaving speculation, and this isn't some impressionistic look at some data with lots of suppositions to fill in the gaps. This has a LOT of data behind it.

Check out Turchin's and Nefedov's book "Secular Cycles" or Turchin's book "Ages of Discord". This AoD page on Peter Turchin's site shows some of the data that he used in that book.

For ordinary people's well-being, he used:

• Employment prospects: (-) fraction of US labor force born outside the US
• Relative Wage: median wage / GDP per capita
• Health: life expectancy and physical height
• Family: (-) age at first marriage

For elite overproduction, he used:

• Top Wealth: largest personal-wealth value / median wage
• Education Cost: elite-university (Yale) tuition / median wage
• Elite Fragmentation: party fragmentation in Congress

He also has measures of various sorts of political and quasi-political violence:

• One-on-one: assassinations (I mention this for completeness)
• One-on-many: terrorism
• Many-on-one: lynchings
• Many-on-many: riots

He mentions in his book two ways of measuring patriotism: naming of states' counties after national heroes as opposed to local and regional ones, and visits to nationalistically significant sites like George Washington's Mount Vernon estate and the Statue of Liberty.

 

[lpetrich]

All three curves - commoner well-being, elite overproduction, and political violence - are strongly correlated:

• 1820's: Peak, Trough, Trough
• 1900's: Trough, Peak, Peak
• 1950's: Peak, Trough, Trough
• (present): Trough, Peak, Peak (where we are headed)

There are also spikes in political violence at 1870, 1920, and 1970, though not at 1820. So this means that the US is headed to a rough time.

There are similar trends in other industrialized nations, even if not as far along as in the US.

 

[rousseau]

God save us from biologists wandering into the social sciences and wondering why we don't just run some simple models. Always with briefcases full of oversimplified analogies and just-so stories.

History is the process where a bunch of academics craft data and books from their university towers, while Trump throws on a Make America Great Again hat and changes the course of the world. The movement of time and history is not rational, it follows ecological patterns and the passions of humanity.

And the law of unintended consequences means that even what we once thought were good ideas, may turn out to be bad ideas later.

 

[T.G.G. Moogly]

This thread reminds me of John Nash's Game Theory.

 

[Politesse]

God save us from biologists wandering into the social sciences and wondering why we don't just run some simple models. Always with briefcases full of oversimplified analogies and just-so stories.

This isn't some handwaving speculation, and this isn't some impressionistic look at some data with lots of suppositions to fill in the gaps. This has a LOT of data behind it.

Check out Turchin's and Nefedov's book "Secular Cycles" or Turchin's book "Ages of Discord". This AoD page on Peter Turchin's site shows some of the data that he used in that book.

For ordinary people's well-being, he used:

• Employment prospects: (-) fraction of US labor force born outside the US
• Relative Wage: median wage / GDP per capita
• Health: life expectancy and physical height
• Family: (-) age at first marriage

For elite overproduction, he used:

• Top Wealth: largest personal-wealth value / median wage
• Education Cost: elite-university (Yale) tuition / median wage
• Elite Fragmentation: party fragmentation in Congress

He also has measures of various sorts of political and quasi-political violence:

• One-on-one: assassinations (I mention this for completeness)
• One-on-many: terrorism
• Many-on-one: lynchings
• Many-on-many: riots

He mentions in his book two ways of measuring patriotism: naming of states' counties after national heroes as opposed to local and regional ones, and visits to nationalistically significant sites like George Washington's Mount Vernon estate and the Statue of Liberty.

Anyone can hand-pick a few statistics and tell an interesting story. Society is a complex organism, and how you attach values to vague concepts like "rise" and "decline" are both potentially arbitrary and realistically, generally grounded in a deeper context of cultural expectations and political aspirations. Shall we go down the list you have there and pick apart each of those values and how they connect to real data sets? Very few of those indices are even really known in terms of tangible and objective data, though estimates abound. :

Like the toponyms; why on earth would we assume that county names are necessarily a strong indicator of "patriotism", whatever that term might be understood to mean? Did he actually arrive at that standard by neutral, logical means? Or is it an indicator that Peter Turchin is an immigrant American, an enthusiastic citizen of a nation which for a host of complex reasons has a lot of counties named for political heroes, most of whom died on various sides of the same civil war? Would it occur to, say, a Welshman, that a strong indicator for his and his neighbors' degree of national feelings might be the number of nearby counties with modern rather than ancient names? Did they get to vote on who to name counties after, for that matter? You can't just blunder into social data sets expecting that whatever first occurs to you as their probable significance is necessarily the most important causative force in their generation.

How would you meaningfully disaggragate visitorship to Mount Vernon between individuals visiting a site of national historical significance as opposed to people who visit it out of interest in Virginian history specifically, or curiosity generated by a personal family collection, or as part of a package tour of "famous sites around DC".

 

[steve_bank]

I listened to an anthropologist talk about why past civilizations failed..

To make a long story short, civilizations are successful and grow based on an initial set of paradigms. In the USA it was 19th century freemarket capitalism

Eventually complexity and scale grows to the point the initial paradigms no longer work. Leadership changes but comes from thje same old cultural pool. Problems mount and are unsolved.

That is where we are now.

 

[lpetrich]

(me on various measures of commoner well-being, elite overproduction, and political violence)

He mentions in his book two ways of measuring patriotism: naming of states' counties after national heroes as opposed to local and regional ones, and visits to nationalistically significant sites like George Washington's Mount Vernon estate and the Statue of Liberty.

Anyone can hand-pick a few statistics and tell an interesting story. ...

That's why one has to be careful about such things, and Peter Turchin seems like he is being careful.

Shall we go down the list you have there and pick apart each of those values and how they connect to real data sets? Very few of those indices are even really known in terms of tangible and objective data, though estimates abound. :

PT himself would concede such difficulties. One has to work with what one can get.

Like the toponyms; why on earth would we assume that county names are necessarily a strong indicator of "patriotism", whatever that term might be understood to mean? ...

Politesse, I recall that you are professionally an anthropologist. You might have run into some discussion of what names people choose for what they name. A common source of names is names of one's heroes.

What sort of heroes would one have if one is a big patriot? National heroes, and for the US, the likes of George Washington and Benjamin Franklin and Thomas Jefferson and Abraham Lincoln and FDR.

How would you meaningfully disaggragate visitorship to Mount Vernon between individuals visiting a site of national historical significance as opposed to people who visit it out of interest in Virginian history specifically, or curiosity generated by a personal family collection, or as part of a package tour of "famous sites around DC".

Mt. Vernon is about 15 miles south of the central part of DC, so one has to make an extra effort to go there. Why go there when one can go to the Lincoln Memorial or the Washington Monument?

 

[rousseau]

I listened to an anthropologist talk about why past civilizations failed..

To make a long story short, civilizations are successful and grow based on an initial set of paradigms. In the USA it was 19th century freemarket capitalism

Eventually complexity and scale grows to the point the initial paradigms no longer work. Leadership changes but comes from thje same old cultural pool. Problems mount and are unsolved.

That is where we are now.

Physics. Any system can't grow infinitely large or small, and must eventually regress to the mean. In the past few centuries we've seen a system of competing states. It's impossible for any of these states to expand indefinitely, and very unlikely that they reduce to nothing. So in lpetrich's example it makes sense that you'd see cycles - something gets big, then eventually it regresses to the mean, then it gets big again.

Being able to quantify this process doesn't imply that we can control or direct the system, though. Even if we could in theory - who would be the 'controller'.

 

[steve_bank]

I listened to an anthropologist talk about why past civilizations failed..

To make a long story short, civilizations are successful and grow based on an initial set of paradigms. In the USA it was 19th century freemarket capitalism

Eventually complexity and scale grows to the point the initial paradigms no longer work. Leadership changes but comes from thje same old cultural pool. Problems mount and are unsolved.

That is where we are now.

Physics. Any system can't grow infinitely large or small, and must eventually regress to the mean. In the past few centuries we've seen a system of competing states. It's impossible for any of these states to expand indefinitely, and very unlikely that they reduce to nothing. So in lpetrich's example it makes sense that you'd see cycles - something gets big, then eventually it regresses to the mean, then it gets big again.

Being able to quantify this process doesn't imply that we can control or direct the system, though. Even if we could in theory - who would be the 'controller'.

Human nature and limitations to cope with complexity.

 

[rousseau]

I listened to an anthropologist talk about why past civilizations failed..

To make a long story short, civilizations are successful and grow based on an initial set of paradigms. In the USA it was 19th century freemarket capitalism

Eventually complexity and scale grows to the point the initial paradigms no longer work. Leadership changes but comes from thje same old cultural pool. Problems mount and are unsolved.

That is where we are now.

Physics. Any system can't grow infinitely large or small, and must eventually regress to the mean. In the past few centuries we've seen a system of competing states. It's impossible for any of these states to expand indefinitely, and very unlikely that they reduce to nothing. So in lpetrich's example it makes sense that you'd see cycles - something gets big, then eventually it regresses to the mean, then it gets big again.

Being able to quantify this process doesn't imply that we can control or direct the system, though. Even if we could in theory - who would be the 'controller'.

Human nature and limitations to cope with complexity.

I agree. I wasn't suggesting that these two things are mutually exclusive.

 

[Koyaanisqatsi]

This thread reminds me of John Nash's Game Theory.

This thread reminds me of the Foundation series and “Psychohistory.”

 

[steve_bank]

Human nature and limitations to cope with complexity.

I agree. I wasn't suggesting that these two things are mutually exclusive.

Chill out...we arejust having an easy exchange.

 

[rousseau]

Human nature and limitations to cope with complexity.

I agree. I wasn't suggesting that these two things are mutually exclusive.

Chill out...we arejust having an easy exchange.

Sorry, didn't mean to come across that way.. it was just a terse comment. Hard to gauge where you were coming from with your post.

I believe on some level a state is like an organism that lives in a system of competing states. In that framework something is eventually going to break down as an individual state gets bigger - in this case very likely human nature and limitations to cope with complexity, and maybe even something resembling the Game Theory mentioned by T.G.G. Moogly.