On this general level, it isn't hard to see the parallels: Trying to optimize situations is something we do every day. On a personal level (fastest way to the restroom), on a group level (best place for dinner), on a national level (unemployment rate) or on a global level (child mortality rate). Optimizing can mean maximizing or minimizing.
The mathematical formulation of the variational principle was inspired by Leibnitz' conjecture that we live in “The Best of All Possible Worlds”. Here, I want to elaborate on the analogy between social and natural systems, and just bounce some thoughts off you.
One should keep in mind that in physics the variations do not “really” take place. They are imagined tries to find the optimal situation, but the optimal solution is then the one which just “is”. Instead, I will here talk about an optimization that takes place over time, where the process of trial and improvement is real.
A Landscape of Possible Worlds
We ask of the systems that govern our lives is that they fulfil a certain task, and help us to work towards a goal. Our economic system is a good example. Its goal is to distribute goods, free capital to allow future investments, and connect traders - all to spur progress. Our political systems have too many tasks to count, but just to name a few: They allow us to live with maximal freedom for everybody without violating anybody else's rights, and they balance long-term with short-term interests. The academic system too has a task, that is to identify valuable research and to build a body of knowledge that helps us to understand nature and actively shape our future.
In most cases however, these systems were not specifically designed and set up for a certain task, they just emerge out of need or out of opportunity and represent cases of spontaneous self-organization. Nevertheless, it is useful to think of a system as having the task of optimizing something desirable, such as minimizing poverty or maximizing the usefulness of education.
For all the possible states the system could be in, consider there is some quantity expressing how good the status of the system is. We will refer to the value of this quantity over the set of possible states as a “landscape” of possibilities (see figure above). In the social context, performance of a system is most often hard to define, and equally hard to measure. However, it might not be necessary to quantify such a function, as long as there are means to evaluate how different states of the system compare to each other. All one really needs to know is whether one situation is “better” than some other, according to some criteria.
It requires effort to bring the system into a more optimal state, since it goes together with an increase of order and decrease of randomness. It also takes constant effort to keep the system in an ordered state. On a fundamental level, many of these efforts are connected to energy supply.
Primary Goals and Secondary Criteria
The essential ingredient to the optimization is now evaluating the system under small changes, to keep the improvements and to toss the worsening. This means the system needs to have means to evaluate how well it is performing. For this evaluation it is first necessary to understand the primary goal of a system, meaning to know what is to be optimized. It can happen easily that during certain phases, aiming for the primary goal goes along with optimizing secondary, derived criteria.
I have previously written about this distinction between primary goals and secondary criteria here and here. It seems to be very common that aiming to optimize secondary criteria instead of primary goals deviates a system from its original task. Bank managers aiming at high bonuses for example might under certain circumstances indeed correlate with an optimal functioning of our financial system, but this correlation is not guaranteed to hold and can lead the system far away from optimizing its primary goal. Similarly, publishing papers is certainly an important ingredient of scientific research and to some extend correlated with research activity and progress. However, the number of publications is a derived secondary criteria that does not generally have to be identical with valuable research. And keeping lobbies happy is a secondary criteria not generally correlated with the primary goal of the political system.
In all these cases, the dominance of secondary goals arises from them being easier to evaluate on an individual level, they thus represent personal incentives. On the long run, pursuing them can lead to a mismatch between the individuals following their micro-interests and the desired macro-behavior of the system, which is to optimize achieving its primary goal. If secondary criteria are used, it is thus necessary to readjust them appropriately so the feedback they provide works indeed towards the primary goal.
Local versus Global Optima
However, when optimizing one has to keep in mind that taking small steps and choosing the direction upwards will lead you to a mountain top, but this will not generally be the top of the highest mountain. A mountain top would be a “local optimum” whereas the highest mountain would be the “global optimum” (see figure below). Improving a local optimum by small changes implies going through a valley.
Yes, this is the same old story with the mountain climbers and valley crossers. Sometimes it has to get worse before it can get better.
Alternatively, one can make a big and courageous change to “tunnel” through a valley. This requires however very precise knowledge about the landscape, otherwise you might end up in a volcano crater lake or something.
Most often, one chooses a combination of both, a first large change followed by corrective adjustments.
For the optimization to work well, there are thus two ingredients needed: variation and evaluation of the variation. Optimization within the system requires the possibility to evaluate the status relative to that of earlier states, and the ability to react to the outcome of this evaluation. The feedback should be such that the system evolves towards an optimum with regard to the primary goal.
Now there are systems in which the evaluation of its performance and the feedback on the system works better, and in others were it works worse. A free market economy for example is a system that detects very promptly consumers satisfaction (evaluation), and is very flexible in its immediate reaction (variation). A planned economy in contrast is very rigid, and if the predictions of the plan are anything off the actual reality, it easily fails to achieve its primary goal. This does not necessarily have to be the case, but given that social systems are very complex and one does not well know what the landscape looks like, predictions are enormously hard if not impossible, and such plans are very likely to fail due to lacking evaluation and variation.
Similarly, governing by monarchy can work to everybodies' satisfaction in case the monarch happens to know well what keeps the people happy and works towards this goal. It is however very unlikely, since the system lacks evaluation and a feedback mechanism that works towards the goal of achieving happiness for the people. A political system that allows for feedback of citizens' happiness has better chances of improving them. Socializing the means of productions on the other hand does not exclude the possibility to evaluating the system's performance, but the reaction to this evaluation is considerably damped due to lacking personal incentives.
In realistic social systems, the system can be pushed out of its limits of applicability in which case it breaks completely down and has to be rebuilt or replaced. For example chopping the king's head off.
Finding an optimal solution also might not be the only thing one is looking for. Given that realistic social systems are subject to statistical fluctuations, the stability of the state is also an important factor of its desirability. If a solution is unstable, it means that a small change can create an even larger change, and drives the system farther and farther away from the optimal situation. This is a so-called positive feedback, though the adjective sounds misleading. A negative feedback instead would want to work against the attempt to make changes.
Keep in mind that it takes effort to keep the system in a well-working state. A positive feedback would for example be one that does not allow anymore to invest this effort, upon which the system retreats to a less optimal state, which means it can invest even less effort and falls into an even less optimal state, etc all down into the valley were the situation is stable again. Dwindling energy supply that corrupts infrastructure and political stability could be such a case.
An additional complication for social systems is that the background one is optimizing in, the landscape, depends on the history of the system itself. Take for example the case in which a major change to improve the political system has been suggested. One took a courageous step, followed by some small adjustment, but the situation turned out to be less optimal than initially. Then returning to the previous state might no longer be possible because the systems' configuration has meanwhile changed. Thus, what previously might have been a locally optimal state might have gone completely lost.
Something else that frequently happens is that mistakes dampen the courage to try changes. One can interpret this loss of courage as deepening the valleys between optima because more people will be unhappy under change. Trying to pass a resolution today might not be the same as passing it next year, even if the content does not change whatsoever. On the other hand, conservatism might increase stability - at least unless the background evolves under you and the optimum you might have been in moves elsewhere. This conflict between the resisting change to achieve stability and the need to adjust to the evolving background is a major tension in our social systems which also reflects in the political spectrum.
This might not be “The Best of All Possible Worlds”, but if we set up appropriately the systems that govern our lives we have a chance to make this world a little better.