Predictions are overrated

Fortune Teller. Image: Vecteezy.

The world, it seems, is full with people who mistakenly think that a theory which makes correct predictions is a good theory. This is rubbish, of course, and it has led to a lot of unnecessary confusion. I blame this confusion on the many philosophers, notably Popper and Lakatos, who have gone on about the importance of predictions, but never clearly said that it’s not a scientific criterion.

You see, the philosophers wanted a quick way to figure out whether a scientific theory is good or not that would not require them to actually understand the science. This, needless to say, is not possible. But the next best thing you can do is to ask how much you can trust the scientists. It is for this latter purpose, to evaluate the trust you can put in scientists, that predictions are good. But they cannot, and should not, ultimately decide what the scientific value of a theory is.

The problem is well illustrated by a joke that my supervisor used to make. He liked to tell his students that whenever you predict something, you should also predict the opposite, because this way you can never be wrong. Haha. In case you are a student, let me warn you that this is bad career advice; You’d also inevitably be wrong, and it tends to be the dirt that sticks. So, don’t /end{advice}. But this joke makes clear that just because a theory makes a correct prediction doesn’t mean it’s good science.

Oh, you may say, you can get away with this once, but then you wouldn’t be able to make several correct predictions. If you said that, you’d be wrong. Because repeated correct predictions, too, are easy to accomplish. In fact, your naïve belief that correct predictions somehow speak for a theory is commonly exploited by scammers.

See, suppose I plan to convince someone that I can correctly predict the stock market. What do I do? Well, I pick, say, 3 stocks and make “predictions” for a week ahead, but that are really just guesses which cover all reasonably possible trends. I then select a large group of victims. To each of them I send one of my guesses. Some of them will coincidentally get the correct guess. A week later, I know which people got the correct guess. To this group, I then send another set of guesses for the week ahead. Again, some people will get the correct guess by coincidence, and a week later I will know which one it was. I do this a third time, and then I have a group of people who have good “evidence” that I can tell the future.

Amazing, no?

What’s the problem here? The problem is that correct predictions don’t tell you whether someone’s theory is good science.

As we have just seen, one of the problems with relying on predictions is that they may be correct just by coincidence. The larger the pool of predictions – or the pool of scientists making predictions! – the more likely this is to happen. The other problem is that relying on predictions makes fundamentally no sense. If I have a scientific theory, it is either a good description of nature, or it is not. At which time someone made a calculation for an observable quantity is entirely irrelevant for a theory’s relation to nature.

This is a point which is often raised by string theorists, and they are correct to raise it. String theorists say that since string theory gives rise to general relativity, it deserves as much praise as general relativity. That’s because, if string theory had been discovered before general relativity, it would have made the same predictions: light deflection on the sun, precession of Mercury, black holes, gravitational waves, and so on.

And indeed, this would be a good argument in favor of string theory – if it was correct. But it isn’t. String theory does not give rise to general relativity. It gives rise to general relativity in 10 dimensions, with supersymmetric matter, a negative cosmological constant, and dozens of additional scalar fields. All this extra clutter conflicts with observations. To fix this conflict with observations, string theorists then have to make several additional assumptions. With that you get a theory that is considerably more complicated than general relativity, but that does not explain the data any better. Hence, Occam’s razor tells you that general relativity is preferable.

Of course, it’s this adding of ad hoc assumptions to fix a mismatch with observation that the philosophers were trying to prevent when they requested testable predictions. But it’s the ad hoc assumptions themselves that are the problem, not the time at which they were made. To decide whether a scientific theory is any good what matters is only its explanatory power. Explanatory power measures how much data you can fit from which number of assumptions. The fewer assumption you make and the more data you fit, the higher the explanatory power, and the better the theory.

Ok, I admit, it’s somewhat more complicated than that. That’s because it also matters how well you fit the data. If you make more assumptions, you will generally be able to fit the data better. So there is a trade-off to be made, which needs to be quantified: At which point is the benefit you get from more assumptions not worth a somewhat better fit to the data? There are statistical tools to decide that. One can argue which one of those is the best for a given purpose, but that’s a fight that experts can fight in the case at hand. What is relevant here is only that the explanatory power of a theory is quantifiable. And it’s the explanatory power that decides whether a theory is good or not.

That’s obvious, I know. But why then do philosophers go on (and on and on) about predictability? Because it’s a convenient rule of thumb. It prevents scientists from adding details to their theory after they have new data, and doing so tends to reduce explanatory power. So, in many cases, asking for predictions is a good idea.

However, if you rely on predictions, you may throw out the baby with the bathwater. Just because no one made a prediction doesn’t mean they necessarily will add assumptions after an observation. In fact, the very opposite can happen. Scientists sometimes remove unnecessary assumptions when they get new data. A theory, therefore, can become better when it has been updated.

Indeed, this has happened several times in the history of physics.

Remember Einstein introducing the cosmological constant and then calling it a blunder? He had mistakenly made a superfluous assumption and then removed it after he learned of the observations. This increased, not decreased, explanatory power. Or think of Dirac’s supposed discovery of anti-particles. When his mathematics revealed a positively charged equivalent of the electron, he argued it would be the proton, which had already been observed at the time. This required the ad hoc assumption that somehow the difference in the masses between the electron and proton didn’t matter. When the positron was discovered later, Dirac could remove the ad hoc hypothesis, thereby improving his theory.

By now I hope it is clear that you should not judge the scientific worth of a theory by looking at the predictions it has made. It is a crude and error-prone criterion. Unfortunately, it has become widely mistaken as a measure for scientific quality, and this has serious consequences, way beyond physics.

Epidemic models, for example, have been judged erroneously by their power to correctly predict the trends of new cases and deaths. But such predictions require modellers to also know what actions society takes to prevent the spread. They require, basically, to predict the minds of political leaders. This, needless to say, is asking for somewhat too much. But, yell the cranks, if it doesn’t make predictions, it’s not science! Nonsense. You should judge epidemic models – any model, really – by how much data they have been able to describe well, and how many assumptions were needed for this. The fewer assumptions and the better the fit to data, the higher the scientific value of the model.

A closely related confusion is the idea that scientists should not update a theory when new data comes in. This can also be traced back to Popper & Co who proclaimed that it is bad scientific practice. But of course a good scientist updates their theory when they get new data. That, after all, is the essence of the scientific method! You update your theory so that it has the highest explanatory power. In practice, this usually means recalibrating free parameters if new information is available.

Another example where this misunderstanding matters are climate models. Climate models have correctly predicted many observed trends, from surface temperature increase, to stratospheric cooling, to sea ice melting. That’s an argument commonly used against climate change deniers. But the deniers then go and dig up some papers that made wrong predictions. This, so their claim, demonstrates that really anything is possible and you can’t trust predictions.

In defense, the scientists say the wrong predictions were few and far between. The deniers then respond – entirely correctly – that there may have been all kinds of reasons for the skewed number of papers that have absolutely nothing to do with their scientific merit.

By now we are arguing about the integrity of scientists and the policies of their journals instead about science. The scientists are clearly losing the argument. And why is that? Because relying on predictions is not a scientific argument. It is inherently a sociological argument. It’s like claiming that a study must be wrong because the lead author has a conflict of interest. That’s reason to be skeptical, yes. But it does not follow that the study is necessarily wrong. That would be a logically faulty conclusion.

What, then, is the scientific answer for the climate change deniers? It’s that climate models explain loads of data with few assumptions. The simplest explanation for our observations is that the trends are caused by human carbon dioxide emission. It’s the hypothesis that has the highest explanatory power.

To add an example that is closer to home: Many non-physicists ridicule hypotheses like supersymmetry and certain types of particle dark matter because they can be eternally amended and hence make no predictions. But that is not the problem with these models. Updating a theory when new data comes in is totally fine. The problem with these models is that they have assumptions that were entirely unnecessary to explain any data to begin with.

Adding supersymmetry to the standard model or details about dark matter particles to the concordance model is superfluous. It lowers the explanatory power of these theories, instead of increasing it. That’s what’s unscientific about it. And of course once you have an assumption that was superfluous in the first place, you can eternally fiddle with it. But it’s the use of superfluous assumptions that’s the unscientific part, not updating them.

In brief, I think the world would be better place if scientists talked less about predictions and more about explanatory power.