Imagine a vast landscape filled with possibilities: new technologies, biological mutations, and scientific theories waiting to be discovered. This “space of the possible” is constantly changing, driven by the forces of innovation and obsolescence.
Recent evidence of the slowdown in technological and scientific progress contrasted with the acceleration of epidemiological risks in a globalized world (in the opposite direction) indicates the importance of relative rates of innovation and obsolescence.
When does innovation surpass, or fail to surpass obsolescence? Understanding this dynamic is still in its infancy, and the way innovation is debated is largely fragmented across all fields. Despite some qualitative efforts to bridge this gap, knowledge is rarely transferred.
In research led by the Complexity Science Hub (CSH), Eddie Lee and his colleagues have taken a significant step towards building those bridges with a quantitative mathematical theory that models this dynamic, offering actionable insights in diverse fields such as economics, biology, and science.
The article, “Idea engines: Unifying innovation & obsolescence from markets & genetic evolution to science,” is published in the Proceedings of the National Academy of Sciences.
The space of the possible and its boundaries
Lee, along with Geoffrey West and Christopher Kempes of the Santa Fe Institute, conceives innovation as an expansion of the space of the possible, while obsolescence reduces it. The “space of the possible” encompasses the set of all potentialities realized within a system.
“Within the space of the possible, one might think of different manufacturing technologies available in companies. All species of living mutations would be a good example in biology. In science, one might think of scientifically feasible and empirically supported theories,” says Lee.
The space of the possible grows as innovations are introduced from the “adjacent possible,” a term employed by Stuart Kauffman to refer to the set of all things one step away from what is possible. Lee and his co-authors compare this to an obsolete façade, which is the set of all things on the verge of being discarded.
The model
The researchers developed a model that tracks the size and composition of this “space of the possible” over time. It considers two opposing forces:
- Innovation: This expands the space by introducing new possibilities, such as cutting-edge technologies or innovative scientific theories.
- Obsolescence: This reduces the space by rendering some possibilities obsolete, such as outdated technologies or replaced ideas.
“One could say this is an exercise in translation,” says Lee, the first author of the article. “There is a wealth of theories about innovation and obsolescence in different fields: from economist Joseph Schumpeter’s innovation theory, to other ideas proposed by theoretical biologist Stuart Kauffman, or philosopher of science Thomas Kuhn. Through our work, we attempt to open the doors to the scientific process and connect aspects of different theories into a mathematical model,” explains Lee, a postdoctoral researcher at CSH.
Three Possible Scenarios
Based on this image of the space of the possible, the team modeled a general dynamic of innovation and obsolescence to identify three possible scenarios:
- Finite space: Imagine a limited landscape of possibilities. Here, even constant innovation will eventually reach a ceiling, and obsolescence will ultimately lead to an increasingly reduced and stagnant space.
- Expanding space: In this scenario, innovation relentlessly surpasses obsolescence, continually adding new possibilities and leading to a constantly growing landscape.
- Schumpeterian dystopia: This scenario, named after economist Joseph Schumpeter, paints a bleak picture. Here, obsolescence surpasses innovation, causing the “space of the possible” to shrink and collapse. Imagine a stagnant world where progress halts due to lack of new ideas.
The model was tested with real-world data from various fields, from measures of business productivity to COVID-19 mutations and scientific citations. In this way, the researchers were able to gather examples that had previously been considered in isolation from each other. Both the model and the data correspond to the average set of dynamics rather than focusing on specific innovations, allowing for emphasized generalization in the article.
“We saw a remarkable similarity among all the data, from economics, biology, and the science of science,” says the CSH researcher. A key discovery is that all systems seem to live around the innovative boundary. “Furthermore, agents found at the boundary of the innovative explosion, whether near or far, share the same characteristic profile,” adds Lee, where few agents are innovative and many are on the brink of obsolescence. West compares this to systems living on the “edge of chaos,” where a small change in dynamics can lead to a large change in the system’s state.
Key findings
The model identifies a critical boundary between expansion and collapse scenarios. Crossing this boundary can lead to dramatic fluctuations in the size and diversity of the “space of the possible,” with periods of both stagnation and bursts of innovation.
But the model goes beyond mere prediction. It aligns with real-world data and provides insights across various fields:
- “Follow the leader” dynamics: The model explains why companies and organisms often follow established concepts until a breakthrough renders them obsolete. Imagine a herd of animals grazing in familiar grassland until a new food source emerges.
- Scientific progress: In science, progress may depend on the obsolescence of old ideas, creating space for new paradigms. Think of the scientific revolution, where the overthrow of established theories paved the way for modern science.
Universal phenomenon
This novel approach could transform our understanding of innovation dynamics in complex systems. By attempting to capture the essence of innovation and obsolescence as a universal phenomenon, the work brings divergent views together into a unified mathematical theory. “Our framework provides a way to unify a phenomenon that has so far been studied separately with a quantitative theory,” say the authors.
“Given the fundamental role that innovation plays in all its multiple manifestations in society, it is quite surprising that our work appears to be the first attempt to develop some sort of grand unified mathematical theory that is testable to understand its dynamics,” says West. “It is still very raw, but hopefully it can provide a starting point for developing a more detailed realistic theory that can help inform policies and practitioners.”
“We provide a mean model of the combined dynamics of innovation and obsolescence,” says Kempes. “In the future, it is exciting and important to think about how this mean model combines with detailed theories of how innovations occur. For example, how do current objects or technologies combine to form new things in something like the recently proposed Assembly Theory?”
Conclusion
The research offers a novel framework for understanding the interaction between innovation and obsolescence. Its testable predictions and interdisciplinary applicability can inform fields ranging from economics and business to biology and science. By understanding the delicate interaction between these forces, we can gain valuable insights into progress, stagnation, and the potential futures that await us.
This research can inform decision-making in various fields:
- Policymakers: They can foster policies that encourage sustained innovation without neglecting sustainability concerns.
- Businesses: They can identify strategies to stay ahead and adapt to changing landscapes.
- Scientists: They can gain a deeper understanding of how scientific progress unfolds and identify areas ripe for exploration.
Reference (open access)
Edward D. Lee et al, Idea engines: Unifying innovation & obsolescence from markets & genetic evolution to science, Proceedings of the National Academy of Sciences (2024). DOI: 10.1073/pnas.2312468120
Note: Prepared with information from the Complexity Science Hub press release and the scientific article.
