Catastrophism abhors a vacuum. As the volume fades on “corporate policies and consumer habits that hasten the destruction of the earth,” it has been cranked to 11 on “AI could kill all humans.” The echoes from 2015 are eerie. “No challenge—no challenge—poses a greater threat to future generations,” intoned President Barack Obama that year in his State of the Union address. At the Paris Climate Conference a few months later, French President François Hollande opened the proceedings by declaring, “never have the stakes of an international meeting been so high. For the future of the planet, and the future of life, are at stake.” Pope Francis released an encyclical.
A decade later, we are doing it all again with artificial intelligence. We have the demands for aggressive regulation, the imperative of global coordination, and the obstacle of an untrustworthy Chinese Communist Party. Pope Leo released an encyclical. And Donald Trump is insisting that predictions of AI doom, like predictions of climate doom, are a “HOAX.”
Of course, neither climate risk nor AI risk is a hoax. But in each case the pitched battle between catastrophism and complacency has left too little room for the pragmatism with which we humans might manage to muddle through, as we typically do. Much of my own early policy work addressed this dynamic in the climate context, analyzing the underlying causes of overheated rhetoric and making the case that one could, and should, accept the physical science of rising temperatures while rejecting the poorly constructed causal chains that led toward “Hell on Earth.”
So my engagement with AI issues over the past couple of years has come with a strong sense of déjà vu. I am wary, going back around this circle again, that perhaps my own habits of mind lead me to reject a prediction of cataclysm wherever it emerges, always constructing elaborate explanation for why things aren’t so bad. This would serve me poorly if cataclysm were indeed imminent. But insofar as a more pragmatic approach to climate change is winning the day, the lessons seem worth applying to our present panic. Yes, we should worry about AI, but how?
Engineers, scientists, and policymakers are accustomed to evaluating concrete and clearly defined problems and proposing directly responsive solutions. But there is another class of problem—what, in the climate context, I called a “worrying problem”—that sends our thinking off the rails. These problems are worrying not only in the sense that they are cause for serious concern, but also in the sense that they are tailor-made for worry. The potential harm rises toward infinity as the likelihood falls toward zero. For the analytical mind, this, quite literally, does not compute.
Identifying a worrying problem’s unique characteristics can help place it in a more useful context and start a more productive conversation about how to respond. Where most problems are immediate, a worrying problem is forecasted. We are not already experiencing the effects, but rather anticipating what they might be. Where most problems are static, a worrying problem is irreversible. Once we are experiencing the effects, it is too late to act. And where most problems are confined, worrying problems are pervasive. The typical policy issue affects some particular social institution or economic sector or region, leaving the rest of our resources available to address or cope with it. Worrying problems operate at the foundational level of society, with potentially interlocking consequences for everyone and everything.
Once defined, worrying problems are recognizable all around us. For instance, while writing about climate change as one worrying problem a decade ago, I highlighted “the prospect of faster computing hardware and more sophisticated software ultimately yielding superhuman and potentially hostile artificial intelligence” as another. Declining birth rates are something of a worrying problem, though they are already here; likewise the national debt, though its effects are not quite as pervasive. A global pandemic is definitely worrying, if no longer just a forecast.
Set climate change and artificial intelligence side by side and both the similarities and differences are illuminating. As a starting point, both are spoken of as broad and poorly specified challenges. Is the actual climate change problem the heat? Sea-level rise? Natural disasters? Food scarcity? Migration? Is the actual AI problem loss of control in the physical world or havoc in digital systems? Enabling bad actors? Military dominance? Labor market disruption? Social dysfunction? The casual or dramatic prognosticator blends all this together and conflates the most worrying element of each. This is going to happen and this could kill us all.
But these aren’t the same this. Even if the concern is “all of the above,” each scenario has a different likelihood and magnitude, which affects how worried we should be, and also different levels of irreversibility and pervasiveness, which dictate our ability to intervene. The scenarios most likely to happen invariably look much more like normal problems—migration into urban areas, new hacking capabilities—that can impose high costs, to be sure, but are well within our frame of experience.
The unprecedented catastrophes, meanwhile, offer opportunities for intervention. One reason that climate scenarios are especially scary is their true irreversibility—say, the accelerating melt of ice sheets and accompanying sea-level rise. But is the core concern really sea-level rise or is it flooding? The latter happens from time to time and has many solutions, especially when the dangers increase so gradually. The change in the physical world may proceed inexorably, but civilization progresses much faster.
The worst AI scenarios tend to represent the flip side of that coin. We may wake up one day and discover that something terrible has happened incredibly quickly, before our institutions or even our engineers could react. The good news is that whatever has happened will almost surely be fixable, albeit at very high cost, and we will learn. We are again back in the world of problems we have seen many times before.
Yes, one can always posit the hypothetical scenario in which the models carefully hide their intentions and presence while propagating across and seizing control of every imaginable device before, in a master stroke, revealing themselves all at once and enslaving humanity. The question is not whether such a superintelligence could exist—stipulate that it could—but whether the technology could leap to that point from a level where it does not even cause conventional-scale disasters without any iterations in between. Realistically, many checkpoints and lesser disasters lie along the way, each offering an opportunity to change course. On the road from OpenAI’s HuggingFace hack to global extermination is a banking system paralyzed, say, or an AI lab sheepishly admitting it has lost access to its own servers and data. These could be enormously costly, and lead to massive human suffering, but they are not existential. They are not even the equivalent of an old-fashioned war.
With climate change, the key question has turned out to be how far in the future are we really forecasting? With AI, it is how irreversible are we really talking? If plausible scenarios exist that skip over the kind of large-scale disaster that typically induces a robust policy response, going straight to a global cataclysm that leaves no opportunity for intervention, they are the ones that deserve greatest worry and costly preventative measures. But that bar is a high one to meet, and the burden is on the worriers to present the account of how their fears might clear it.
A common dynamic across worrying problems is that the worst potential effects are mediated through numerous social and economic layers that do not behave in the rigid ways modeled by scientists and engineers. The layers represent, assumed away in the technical analyses, present opportunities for action. The climate may yield hotter temperatures, but people may install air-conditioners. Precipitation in some regions may fall, but people may develop drought-resistant crops.
Likewise, AI models may empower hackers, or even pursue hacks themselves, but the ability to hack has always existed and the hacks now generated by AI are of the usual type. Successful hacks only accomplish so much. And the same AI tools will also strengthen cybersecurity. The reason we do not see more frequent and sophisticated terrorist attacks is not a lack of access to technical capabilities, but rather a lack of competent actors with rational motivations and organizational capacity. Great power competition can accommodate large imbalances in military capability and that competition also tends to close them quickly.
Economic limits also impose bottlenecks on runaway scenarios, most obviously in the case of labor-market disruption and mass unemployment, where the rate of technology adoption is never anywhere close to what technologists anticipate as they demonstrate new possibilities in their labs. “We’ve all been too ambitious on timelines,” Sam Altman acknowledged last month. “Even with this incredible technology, society and the economy will adapt more slowly.”
That constraint is relevant not only for economic impacts, but also for deployment of the “physical AI” (robots) that is often a prerequisite for doomsday. In the Bay Area, one can find serious researchers predicting that robots will replace all human labor in the next five years. Where would we even get so many robots? The robots would build them. Sure. Anthropic’s Economic Scenarios for Transformative AI offers a useful reality check. In its “extreme” scenario, GDP growth averages roughly 9% per year through 2030. That’s an extraordinary economic transformation, but it does not suggest a state of technological advancement in which production systems have been so rapidly and completely remade that autonomous robots are building autonomous robots at world-conquering pace.
The experts and activists, focused on the technical reality of global temperatures or LLM capabilities as a direct proxy for risk and damage, see the only response as mitigation. In the climate context, this means reducing greenhouse gas emissions to slow the rise in temperatures. In the AI context, it would mean slowing the progress of model capabilities. Those policies have their place. Moving to lower-emissions technology has many benefits. So does advancing AI research at a speed that labs can understand and manage.
But for the most part, the mediating layers between the technical reality and the human reality offer the best sites for policy intervention through adaptation. The exception that proves the rule, a meteor strike, presents the case where adaptation is impossible and mitigation imperative; humanity should be investing much more to ensure foolproof detection and deflection capabilities. (Maybe with AI we will do so.)
But in the climate context, as we improve monitoring and forecasting, and as we develop technologies and upgrade infrastructure, we are coming to recognize the problem as both real and manageable. In the AI context, regulators should absolutely monitor frontier models in the labs, just as they monitor risk in large financial institutions; labs should be liable if their models behave badly, just as manufacturers face strict liability for their products; and dangerous incidents should be investigated, with immediate halts pending fixes, just as aviation disasters are handled today.
We will need to harden digital infrastructure and deploy the most advanced AI tools to detect and interdict threats. The technology will also change society in various ways, as technology does. Our laws and norms will adapt, as they do. We could even retreat a bit from total, society-wide reliance on digital control of everyday life (though perhaps this should already have happened at “smart beds flipped out during the AWS outage, and so did their sleepy owners”).
These sorts of proportionate responses—monitor the labs, impose liability, investigate accidents, harden infrastructure—are hardly novel. They are exactly what cooler heads, not to mention many of the catastrophists, already propose. And that may be the most telling analogy of all.
The apocalyptic climate rhetoric from activists and politicians, and their dreams of a world-war-scale national mobilization, were always hard to square with their milquetoast policy proposals. Likewise, few of the people warning of AI doom seem much inclined to ban GPUs, shut down data centers, threaten world war, or do any of the things that would immediately become rational if we faced a 10% chance of near-term human extinction. No, where the rubber meets the road, they mostly want to approach the problem as a normal one, in which case, what work is the catastrophism doing besides attracting attention?
Conversations will be most productive if we keep the proper perspective, talk concretely about what may happen in reality, and hold on to a humility and confidence derived from recognition that we live in, and have managed to flourish in, an unpredictable world. The immediate challenges that we face are addressable, and the forecasted ones will arrive in ways that provide time to react, and correct course, so long as we are paying attention. The truth, it turns out, is often more convenient than advertised.
I think this article is a helpful reminder that we should spend more of our energy working out responses to the practical problems of these issues rather than wasting energy in paralyzing doomerism. That said, I think it's important to recognize that one cause for doomerism is the sheer scale of potential impact. For example, the threat of sea level rise is not merely isolated floods, for which humans have contended with for thousands of years. It's that sea level rise simultaneously threatens a disproportionate share of the world's populations, largest cities, economic engines, and financial systems over a sustained period of time (unlike, say, hurricanes). The problems themselves (flooding, migration, etc.) are not necessarily novel, but it's the possibility that they arrive at a scale that overwhelms our capacity to respond. In the long run, humans will adapt and carry on, but also in the long run we are all dead.
And I will say take Americans' fears seriously and try to convince them through rational discourse that their fears may be overblown.