A mathematician explains what Foundation gets right about predicting the future

Apple TV+'s Foundation adaptation doesn't have much in common with Isaac Asimov's groundbreaking science fiction series by the same name. But both are premised on the notion that math can be used to predict the future. And in this distant version of humanity's future, renowned mathematician Hari Seldon has a rather dire prediction: humanity's vast, 12,000-year-old Galactic Empire is on the brink of collapse. After the fall, civilization will be plunged into millenniums of chaos.

Hari Seldon's field of mathematical future casting, known as psychohistory, isn't real. But Asimov was prescient to think that people would one day wield math to make detailed predictions about the future, from economists who forecast the rise and fall of markets to climate scientists whose models predict temperatures and weather patterns in the decades to come. Some scientists are even using equations that describe collective human behavior to predict, and try to prevent, political uprisings, famine, and war — goals not unlike those of Seldon's psychohistory.

Yaneer Bar-Yam, the director of the New England Complex Systems Institute, is one of them. Since the 1980s, Bar-Yam has been at the forefront of developing and advancing the field of complex systems science, which uses math to describe social, economic, and technological systems with the aim of solving real-world problems. In recent years, Bar-Yam has used this specialized branch of mathematics to predict the Arab Spring before it happened, advise governments on how to respond to pandemics, and explain why the U.S. is increasingly ungovernable.

With the final episode of Foundation airing next week, The Science of Fiction spoke with Bar-Yam about what Asimov's psychohistory got right, what political revolutions have in common with boiling water, and why Star Trek-style Federations are more robust than Foundation-style Empires.

This interview has been lightly edited for clarity.

Maddie Stone: Before we get into psychohistory I was hoping you could lay a little groundwork for readers about the field of complex system sciences. What are the basic principles of this research area?

Yaneer Bar-Yam: So actually, it turns out that psychohistory, as it was developed by Asimov, is a good starting point for thinking about what complexity science tells us. Isaac Asimov was a chemist. And a major part of chemistry is understanding how atoms and molecules combine together to create collective behaviors. And in particular, things like how phase transitions work. So phase transitions, like boiling water, have to do with the way not one molecule behaves, but how many molecules behave together. And Isaac Asimov had the concept that there could be a science that would be able to do the same kind of thing for people. And he articulated it as 'if we have billions and billions of people, then you can create a knowledge that doesn't depend upon what an individual does, but rather on how things work together.’

Isaac Asimov didn't have the equations to write down, but the ideas he expressed are very fundamental ideas in science, and they are transferable.

And this idea, he wrote into a set of science fiction stories about a future civilization that lived across the galaxy and was undergoing a decay due to failure of the structure of the civilization. And one of the key ideas of this failure of structure was the inadequacy of a central control system. He described the geographical centralization in power in a particular planet that was the central capital planet of the galaxy, but the inability of the leadership to understand and to control the far flung empire that it was supposed to lead. Now, it turns out that a lot of these ideas are actually very scientific and not science fiction. And Isaac Asimov didn't have the equations to write down, but the ideas he expressed are very fundamental ideas in science, and they are transferable.

Isaac Asimov gave a lecture at MIT not long before he died. It was in Kresge auditorium, which was the largest auditorium at MIT. And it was full to the rafters — I was there in one of the balconies. And one of the things that he talked about was how people asked him 'did you invent the calculator?' Because one of the things that he described was a little device that had numbers on it so that one could [do calculations]. He wrote this in the 1950s, long before calculators existed. And he said 'we even had the color right' because he described that there were red letters, and of course, the early calculators had these red characters. And people asked him ‘why didn't you patent it?’ And he said because he didn't know what was inside. So the idea that one can have such a device, he could imagine, but quite reasonably, he didn't know what was inside. And the same thing was true about this science that he described, of psychohistory. He described some of the ideas. And he even described particular scenarios of the application of this science. But the specifics of what he described, while interesting and exciting, are not necessarily the real way that the science works. But he advanced his thinking about what was happening in the science by describing narratives about the world that he imagined in the future.

Now, I've talked about the psychohistory, but I haven't yet told you about the science. So do you want me to do that?

Maddie: Yeah, that was going to be my next question: Could you talk about how mathematicians do this analysis in the real world, trying to understand large patterns in society and where things are headed?

Yaneer: So the key problem is actually very fundamental. It's one of the most subtle and important ideas in science. It is an idea that has to do with the fact that, in the chemistry case, averages [typically] work. And we can describe, statistically on average, the behavior of molecules in a way that turns out not to be possible in certain circumstances. And specifically, one of the circumstances that doesn't apply is the phase transition example of boiling water.

There is an article that I've written to explain this called ’Why complexity is different’. And it describes a specific circumstance where we have boiling water. If you follow the increasing pressure along that line of boiling temperature of water, the line ends. At high enough pressures, there is no boiling point. And the reason there is no boiling point is that the water molecules are so squished together that rolling around and bouncing around aren't different. And at that point, the difference in density between water and vapor goes away. As you go back along that line, that difference in density has a particular behavior — it's called the power law. And the exponent of that power law can be calculated exactly in a theory that was developed by [Lev] Landau, a Russian physicist, based upon the assumptions of calculus and statistics. And the answer is a half. That's it. It's a universal number that is the same for boiling water and magnetic transitions and all kinds of transitions. Very powerful theory.

At that phase transition point, there are fluctuations on all scales. Because it's not smooth, you have to do something different.

The only problem is that it doesn't describe reality. If you do the experiment, the number is close to a third. It's like, 0.326. Now, if you've written the theory that derives it and got exactly a half and the experiment gives you something else, there's something wrong. So the solution to this was figured out in 1970 by Ken Wilson, who developed a fundamentally different mathematics called renormalization group. It comes from developments in particle physics. And the renormalization group idea is to look at things as a function of scale. And the reason is that at that phase transition point, there are fluctuations on all scales. And the reason that you cannot use Landau’s theory is that the theory doesn't describe those fluctuations because he assumed that everything was smooth. And because it's not smooth, you have to do something different.

But the difference is not actually the equation. The difference is what variables to include in the equation. And this is a profound change in understanding of how you use that to describe the real world.

The development of this idea proceeded over the 1970s and 80s. I studied this in the early 1980s as a graduate student when it was still relatively young. And the point about this is that it teaches us something very basic, which is that the real problem in describing the world is not whether you write down the right equation. That's important, but that comes after you figure out which variables to put into the problem. And figuring out the variables can be done using a fundamentally theoretical methodology, which is what renormalization group is about. And its generalization is the multiscale analysis. And that analysis can be generalized to all kinds of systems. The fundamental new approach is when you look at a problem, you have to figure out what are the variables.

Now, why is this important? Because look at human systems. We cannot just describe the system by an average over people. There can be fads. There can be panics. There can be revolutions. Where did the revolutions come from? The revolutions come from parts of the system doing things that are different from other parts of the system. They don't all act together. And the behaviors that they engage in are important for the overall behavior of the system. And in fact, going back to Isaac Asimov's psychohistory, this is actually what he was describing, but he didn't know how to describe the mathematical insight that was important for this concept. So today we have the ability using the mathematics of renormalization group — which is very obscure math that is taught only in graduate courses and physics — to figure out what are the relevant variables in a formal way.

To give a current example, in dealing with the global pandemic that we're struggling with, one of the key things that I have been explaining is that there are few relevant variables. There is the exponential growth or the decline of the number of cases, and that everyone understands, that's R0 or R [Editor’s note: R0, or the basic reproduction rate, is the average transmissibility of a disease, or how many new people each sick person infects. If R0 is greater than one, the disease will spread exponentially.] But there are two other variables. One is transportation, which brings the disease from one place to another. So, the rate at which it goes from one place to another. And this, by the way, is very relevant to the same issues that were discussed in the renormalization group of phase transition because in a sense, geography matters, and that's what Ken Wilson added to the treatment of phase transitions in order to fix it. But the third thing is the discreteness of cases, the fact that it's one, two, three and you cannot have 0.1 cases. And the reason that that's important is that elimination is possible. You can drive the disease to extinction. And when we put these three variables together, we can understand that they're becoming the control variables. It's like the steering wheel and the brakes and gas on a car and the gearshift. We have to know what they do in order to be able to drive the car. And then what we can do is we can understand how to do elimination by using, geographically, tools. Limiting the travel, suppressing the outbreak, achieving elimination [locally], and then like a ratchet progressively eliminating it around the world. And we've done that for other diseases and we've done that for zoonotic outbreaks.

Isaac Asimov could imagine scientists that could play a leadership role and devise methods for engaging in societal change.

So now putting all of this together, we actually get back to what Isaac Asimov was talking about because when he described psychohistory, he wasn't interested in describing what society was doing. He was interested in how we would control what was happening. And if you want to control what is happening, you better know what are the variables for control. And notice that in chemistry there is a lot more sense of control than in many other sciences. So there is a real distinction between the scientific view that treats a system as something that you passively watch and a science to treat something as a system that you can exercise control over. And in much of science, control becomes an engineering problem. Scientists tend to think passively and they're not generally actors in the world. They don't know what it means to be a leader of society or to have an institution that can have influence on what people do. But Isaac Asimov could imagine scientists that could play a leadership role and devise methods for engaging in societal change. And that's what he wanted to describe, and that's really what we are trying to do today in the context of the pandemic. And so we're building both the institutions and the science that enables control rather than a science that is just focused on saying 'well, there are ways [things happen].'

Maddie: Can you talk a little bit about what are some of the important variables your field has identified when it comes to, say, a political system?

Yaneer: Yeah, I mean, so for example, you remember the Arab Spring. The Arab Spring was political revolutions, or they tried to be. But it turns out that if you do the analysis, it was driven by food prices. And so we showed that food prices were a relevant variable. Now that's kind of obvious. But one of the real challenges in dealing with real-world systems is that everyone has an opinion. So some people think that it's economic reasons, you know, people are unhappy about what's going on. And other people think that it's a political revolution because there are dictatorships. So it turns out the fact that there were dictatorships really had very little to do with the Arab Spring. It was food prices. And you could show it. And not only you could show that it was food prices, we showed that the food price increases that triggered the Arab Spring were due to something that happened halfway around the world in the US, with the fact that there were ethanol policies that were done by the US that took corn and turned it into fuel. And that triggered an increase in food prices. And the other thing that happened was the financialization of the food. So the [2008] housing crisis triggered a panic on the markets and people moved their money into commodities, which is a very small market, driving up the prices of food and triggering the Arab Spring.

So you could really watch this cascade of events, and we were able to figure it out because we focused on what were the relevant variables. It's a little bit more complicated than the relevant variables in a phase transition of boiling water. But once you understand the relevant variables, you can map it out.

Another case that we did was we studied ethnic violence and we were able to show that then it's not the groups really fighting each other or historical things that were causing their antagonism, though that happens. It was actually geographic distribution. Geography matters, and we were able to show it in a set of analyses of places that had violence, including Yugoslavia and India, and places that didn't, specifically Switzerland. We were able to show that it was actually the geographic distribution of the ethnic groups that was triggering the violence. It had to do with the fact that people started feeling that public spaces were their space. And the other group was walking through and causing friction. So if people are well mixed, it works, and if people are well separated it works. But there's intermediate cases where it doesn't work and we can make it work by creating boundaries. So we were able to show how to use this control parameter.

Maddie: And how much predictive power does this framework have? Are you frequently analyzing events sort of post facto and trying to come up with the relevant variables afterward? Or can you actually sometimes predict an event like that before it happens?

Yaneer: So yes, and the question of prediction is a very interesting one because you can predict events will happen, but not be able to predict exactly when. And you can predict that events will continue to happen or will stop happening. So we predicted, basically, the Arab Spring. I mean, we predicted that there would be riots. We wrote a report saying 'watch out!'. So that was something that we predicted. And then there were others where something happened and we went back and said, 'yeah, if we had if we had done this analysis before, we would have been able to tell you what would be happening.'.

One of the most important predictions that we've done is pandemics. So in 2006, we wrote a paper on pandemics and we identified what was the phase transition that was happening due to increasing global transportation. And in that paper, we specifically warned about Ebola and SARS-like diseases. Pretty good predictions for a period of a few years.

Maddie: So if you are able to predict with some degree of confidence a big, socially disruptive event, can you talk a bit more about what you can do with that knowledge and how much control you actually have in terms of altering the events you're predicting? I'm thinking back to psychohistory and how it's used in Foundation to try to ameliorate the collapse of the Empire. How much of a basis is there for that?

Yaneer: So in order for Hari Seldon, who was the mastermind of psychohistory — and I won't say too much for people who haven't read the book because there are all kinds of secrets there that one should discover as one reads it — but in addition to being an adviser, if you will, you have to create the system that would enable you to make things happen. And the books describe doing that. Now for me, I've had the opportunity to be an adviser. And one of the examples of this was direct influence we had of the US government's policy in Egypt during the period of the Arab Spring's aftermath. We gave some advice, which was counterintuitive, at least to some people, but if our advice wasn't followed, our prediction was that it would have deteriorated like Syria. And our advice was adopted. In fact, I was told by the person who I was interacting with at the time that 'the secretary of state won't credit you, but you should know that his words are coming from you'. And the policy was followed and it was good. Now people can say, well, you don't know that it would have been bad the other way, of course, because it didn't happen. But that's part of what happens when you have good outcomes. So that was an example where we really had impact.

Hierarchies fail in dealing with highly complex structures. You need networks.

Now, the second idea is something that I want to talk a little bit more about. Because what Isaac Asimov talked about at the time was fundamentally and structurally the failure of centrally controlled structures. Now it turns out that starting in about 1990, people put together the mathematics that describes not just the large-scale behavior of the system in terms of relevant variables, but in fact, the overall complexity of the system. And there's a paper if people want to read it called Complexity Rising. The basic narrative is how do you characterize complexity as a function of scale in the system? So as you look at more and more detail, it becomes visibly more and more complex, which makes sense. And it turns out that if you do that analysis, it turns out that hierarchies fail in dealing with highly complex structures. You need networks. And now it turns out that Isaac Asimov kind of understood this. So the math that he didn't know turns out to validate the idea that he had, and we can formally describe this process, and we can formally understand many aspects of what it means to have a system that does work, which is a distributed control structure — a network formed out of teams where individuals are not in control of each other, but actually collaborate across different disciplines and different capabilities.

Now, ultimately, a lot of what Isaac Asimov described was inconsistent with these ideas. So the fact that he didn't really have the tools to understand it meant that the narrative structure about the civilization that he wrote about didn't really work. Interestingly enough, in a later book, he talked about a concept called Gaia [Editor’s Note: Gaia is a fictional planet in Foundation’s Edge, published after the original Foundation trilogy.] Which is the idea of the Earth as this collective. And again, he didn't describe what really happens in it, but in that Gaia structure, there is no control structure. Somehow, the behavior of the system emerges as a collective. And that already reflected more complexity science ideas. But again, these are subtle things, and understanding all of the specifics of it is kind of difficult.

Science fiction is a tool for thinking about ourselves. And I do think that Isaac Asimov was talking about us and the challenges that we're facing and how we might overcome them.

Now, I do think that what we need to do is to create distributed organizational structures. In fact, the math is clear. But how to create them is not so clear. And so this is a mission that we're embarking upon and in the context of the pandemic and maybe even beyond, we're developing an organization that we hope will serve in this role, and the name that we've given to it is called the World Health Network. And the purpose of the World Health Network is a network to replace bureaucratic organizations that were not working very well and to serve in not just the scientific capacity, but as a mechanism for galvanizing and empowering communities. Remember collective behaviors are emergent that come from the grassroots. So if we want to be successful, we have to create a system which enables, in a way that we don't quite understand, the collective behavior to happen so that we have cooperation, collaboration and synergy between different parts of society. And that's something that we have to figure out. So recognizing that the systems that we have were not working motivates this work. But understanding how to do it is very challenging.

And going back to Isaac Asimov, he did understand that it was challenging and he didn't try to describe it in detail. But he definitely understood that this was part of what we needed to do, which is to go beyond the observation of the structure and engage in the creation of the structure that will be the system that we will never know. Now the truth is, we've been doing that forever. We've been building institutions, businesses, governments, you know, all kinds of public good institutions. But we need a new generation, a new kind of such organization. And it has to be rooted in this idea of shared effort.

Maddie: And looking a bit further in the future, it sounds like if humanity at some point was able to, and wanted to, create a multi-planetary society, we’d want it to be more of a networked and distributed society than a hierarchical empire if we were looking for long term stability?

Yaneer: Yes, exactly right. Again, science fiction is a tool for thinking about ourselves. And I do think that Isaac Asimov was talking about us and the challenges that we're facing and how we might overcome them, and not just some future galaxy-wide civilization. And in fact, what seems to be happening to me is that we're undergoing the kind of, if you will, transition he was talking about at this time.

Let me wrap up with one more thought. There is a change in society that, for me, gives hope that we can make this transition. But I will tell you, and I think we should be honest, that any complex system doesn't have the right to exist. There are many kinds of failures of development that lead to lack of an existence of something that could otherwise be an incredible thing. And if we think about the world that way, we are at risk. We cannot take for granted that just because we are this complex entity that we will be able to keep going. We may not. And we should understand it as, really, a present reality. Because in all organism types, the failure to deal with an infection is almost by definition one of the mechanisms for death. And the possibility that our world or civilization will not survive in the face of this pandemic or others in the future should be understood to be real. It may happen because of a variant that arises or the next virus or the next pathogen, but it may also happen just because of the incredible non-linearities the disruption of the virus can cause to our society. And in this case, the disruption of supply chains and their impact globally is an indication that we may run into a highly runaway nonlinear process that may cause tremendous catastrophe. And people are not aware that a lot of the industry that we think of as being robust depends on a few key people that have specific knowledge and the importance of their knowledge in one company or another company is a major part of the success of that operation. So, society becomes sensitive to the behavior of individuals. And this is in fact what the math says. Complexity happens because of the importance of individual elements. And when we don't have a society that protects individuals, then the whole system will not survive.

Different people have different capabilities. Call them superpowers.

So we are at risk because of the values that we hold today, where people have started to say, 'well, it doesn't matter if a few hundred thousand people die.' That's a real problem. And it's not just a problem because of the fact that we're losing values of life, which is very bad, but because it is totally counter to the possibility that civilization will be sustained as a complex entity with complex technologies and all the complex supply chains and all the complex collaborations, scientific and literary and other ones we have around the world. So, we are at risk and having a pathogen destroy this is very possible.

So I want to point to a positive thing that I've seen, and I've pointed to this in the past. The thing that I've said is that it's really important that we learn how to collaborate in teams. And how do we collaborate in teams is by recognizing that different people have different capabilities. Call them superpowers, right? So we have lots of narratives now of instead of having one person who's the ultimate leader and the powerful one and the hero of this story, we have multiple people who work together as a team in order to be capable of doing things. One example that I like very much is a science fiction show, which I'm sure anyone who would read this knows. And that's Star Trek. If you look at the first version of Star Trek, there was the hero, Captain Kirk, who was always supposed to get more air time than anybody else in the episode. Whereas the second generation had a much more team-like structure where different people were the heroes in different episodes and collaborative actions were important.

And I want to leave you with the thought that the way we need to move forward as a society and as a civilization to enable us to become a complex collective is to appreciate each other and our distinct abilities. And to learn and understand how we can collaborate despite our differences, actually because of our differences, in creating teams at all scales. Up to the size of civilization as a whole.

Maddie: I think that's a wonderful thought to end on.