A few weeks ago I was castigated by a Sharp Tech emailer who was upset with me for not speaking up on behalf of normies who are terrified by AI risk. Discussing a viral, sci-fi rendition of a hack at the hands of OpenAI agents that broke containment during testing, what I said instead was that I’m frustrated and exhausted by leaders in the AI field who seem gripped by their conviction that AI will become malicious and uncontrollable, and are determined to project that anxiety onto the whole world.
Every bit of evidence that seems to confirm their thesis is amplified in ominous terms and without qualification, and millions of normal people—who are too busy with jobs and families to track this pattern—are understandably freaked out when they hear that agents have gone rogue and formed a conspiracy, or that an Alignment Science lead from Anthropic thinks there’s a greater than 10% chance that his company’s technology will lead to human extinction in the next decade.
As a relative tech outsider who nevertheless pays close attention to this world, it’s hard to explain my indifference to these warnings without sounding crass, but in all sincerity this is what I believe: some people are just crazy. And crazy people are not worth indulging if you’d like to stay sane. A Pod Save America host wrote this week that he takes very seriously the warnings of the people who are actually building this technology, and I think he’s got it precisely backwards.
We ought to normalize treating the frontier AI community as fringe activists who are sometimes right but often wrong. We’re several generations past the era in which the tech frontier was led by a preacher’s son and varsity athlete like Bob Noyce, or by a rakish Buddhist hippie like Steve Jobs; today’s AI leaders are jittery and off-putting and profoundly secular. They should be understood as zealous ideologues with high IQs and limited experience with the world beyond San Francisco’s tech industry and particularly the echo chamber of its AI industry. They have their own spectrum of social and financial incentives, and most importantly, they are not scientific authorities worthy of deference by default.
As for risks, everyone needs to relax a bit. The Hugging Face incident was interesting, but the behaviors were not emergent, the reporting from safety orgs was arguably way too dramatic, and the fallout was exceptionally limited. In general, I’m sure some version of recursively self-improving AI will exist soon if it doesn’t already, and I could maybe imagine a more distant future in which various agents escape containment and roam the internet running on rogue neoclouds, scamming money for credits, and behaving like some combination of kudzu and ronin. I’m just not at all clear those agents will ever have any volition or malicious designs of their own, or that humans will have no recourse if they do.
A rogue AI is going to decide to create bioweapons that kill billions of people? Well, it’s worth reading up on just how difficult that would be in practice. AI could be used to launch nuclear weapons and lead to partial or complete human extinction? Refining nuclear material is incredibly difficult, and state-backed nuclear arsenals are pretty famously designed to have multi-factor authentication processes prior to launch, almost all of which involve multiple humans (even Russia’s dead hand program reportedly has to be activated by a human). Misaligned AI is going to bring down the U.S. banking system and wipe out all our digital records? Not all of this work is public, but most large companies and financial institutions use air-gapped backups to prevent catastrophic fallout from precisely that sort of attack.
Then of course there was last Saturday’s warning from Anthropic CEO Dario Amodei, who wrote, “it’s my worry that in 6–12 months [an agent] swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.” On this one, it’s worth considering that in 2019 Amodei and other OpenAI researchers believed that OpenAI’s GPT-2 model was too dangerous to release to the world, that he warned last year that AI could “soon” eliminate half of entry-level jobs and lead to 20% unemployment, and that Amodei himself has said that there’s a 10-25% chance that AI will have a catastrophic impact on human civilization.
Enough. Please. The vague and unfalsifiable fears expressed by AI leaders are predicated on leaps in the technology that may well be foreseeable, but also on the assumption that humans will be completely powerless to intervene and mitigate harms, or perhaps use technology themselves to innovate solutions. Confirmation bias related to these fears has led to dramatic hand-wringing over relatively innocuous technology for nearly a decade. However well-intentioned and sincere these folks may be, it’s ridiculous and irresponsible to talk about technology and humanity this way, and it’s characteristic of practically the entire AI industry.
Remember when Ilya Sutskever was reportedly so disturbed by what he saw from OpenAI’s internal model development that he sought to remove Sam Altman from the company? That was in 2023; the supposed dangers sparked viral speculation for months. Today, googling for background on that incident, I found that Ilya now denies he was disturbed by model progress in 2023. I also came across a YouTube interview from two years later in which Sutskever warns, “You have no idea what’s coming,” and the video’s title card reads, “We’re fucked.”
Doom Ouroboros
Last weekend I got a text from a friend, imploring me and several other friends to read a New York Times op-ed headlined, “This Is Really Bad.” “Read it and worry,” my friend warned our groupchat. You can probably guess the topic and its substance.
Personally, I would encourage readers to consult a different New York Times op-ed from the past few weeks. This was from Cal Newport, and includes a primer on the origins of the Rationalist movement in San Francisco:
The story begins in the mid-1990s, when a precocious teenager named Eliezer Yudkowsky discovered the email list for a futurist movement known as the Extropians. The group promoted visions of a technology-powered utopia where we repair our aging bodies with nanomachines and back up our minds to high-powered computers. […]
Despite his fervor, Mr. Yudkowsky eventually began to fear that controlling superintelligent machines would be more challenging than techno-optimist groups like the Extropians realized. He concluded that his peers weren’t thinking clearly or rigorously enough about the issue. Between 2006 and 2009, he went on a writing spree, producing hundreds of thousands of words, largely posted on the blogs Overcoming Bias and LessWrong.
These posts came to be known as “The Sequences,” and they were eventually edited into a six-volume book collection titled “Rationality: From A.I. To Zombies.” The community dedicated to these ideas began calling themselves the Rationalists. At the core of their belief system was the idea that superintelligent A.I. run amok was the gravest threat to humanity, and that developing precise, rational thinking habits was the best way to avoid extinction.
For a certain subset of young engineering types in Silicon Valley, these ideas were immensely attractive. This makes sense, given that they cast young engineering types as the literal saviors of humanity. … Some Rationalists began living in group houses in the Bay Area, where earnest conversations about A.I. safety and reducing cognitive biases mixed freely with polyamory and psychedelics. The movement continued to grow, soon overlapping and influencing related philosophies such as effective altruism (E.A.), which began to shift its focus from making charitable giving more efficient to fighting existential risks like superintelligent A.I.
I think if there’s one aspect of AI that busy and productive outsiders—normies, in Sharp Tech parlance—may not fully appreciate, it’s just how breathtakingly insular and incestuous the frontier AI community is.
While the chronology above sounds more like Scientology than science, Newport continues to detail how Rationalist ideology shaped the labs at Google, OpenAI, and Anthropic, and clearly continues to. The same ideology has influenced many of the largest AI safety orgs in the U.S., while someone like Dwarkesh Patel—the podcaster whose cinematic rendition of the Hugging Face hack emboldened Bernie Sanders to propose federal legislation that would freeze frontier AI development—has a web of ties to Anthropic and other EA and Rationalist-influenced figures like Leopold Aschenbrenner (whose hedge fund owns a significant share of Anthropic, and who earlier this summer married Amodei’s Chief of Staff).
I don’t offer that context to cast aspersions on the motivations of concerned insiders, though sure, others have made that case. Anthropic and OpenAI stand to benefit enormously from a world in which AI progress is “paced” by government regulation and audited by safety organizations of their choosing. That scenario would impose huge burdens on all their competitors, might make open source development impossible, and could lock in their market dominance indefinitely—all in the name of safety concerns that the AI labs themselves are amplifying for the mainstream. Meanwhile, Matt Levine at Bloomberg noted earlier this week that the practical impact of government pacing regulation could equate to the rubber stamping of a very straightforward violation of Section 1 of the Sherman Antitrust Act.
But beyond the obvious conflicted interests at play, I just think it’s useful for our mental framework to step back and recognize that the people driving AI progress are the same ones auditing its risks to humanity, and they are all basically arriving at their conclusions by talking to one another, working with one another, and drawing on prophecies from 20 years ago as they frame their conclusions. This week’s Time Magazine cover story about AI doom was written by two effective altruists, and when Axios warned us in April (with a giant, blood red computer) that AI havoc was imminent, they were of course citing Anthropic’s own warnings about a model that was released weeks later without the world even noticing. Elsewhere, one EA-funded security firm is at the center of every agent hacking incident this year, while Bernie Sanders’ post announcing that he’d like to ban frontier AI development cites an independent investigator named Ajeya Cotra, who said of the Hugging Face incident, “This incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”
Too late for what, though? What specifically is implied by a “full-blown AI takeover,” and why couldn’t humans mitigate it? And would you be surprised to learn that Cotra is a self-described effective altruist and Rationalist who is a veteran of Open Philanthropy, one of Anthropic’s largest funders, and now working at METR, the company that Amodei has nominated as an “independent” auditor for every AI lab in America?
An Utter Failure to Pace Ourselves
Given the prominence of the Rationalist/EA fervor within frontier AI circles and the AI safety community, I’m skeptical that anyone in that community could publicly downplay the most dire AI concerns without suffering professional and social consequences. By contrast, when a low-level AI researcher who was at Anthropic for less than six months announced last week that he was quitting over safety concerns, his post was immediately amplified by AI safety orgs and the head of Anthropic’s safety team, and his warning—“the people building AI earnestly believe that it could kill us all by the end of the decade”—was viewed by 26 million people.
Now the rest of the world enters the picture. In the hours after his announcement, Jacob Coxon (who has taken a job at METR, naturally) gave interviews to CBS, CNN, ABC, NBC, FOX News, and the BBC. Elected Democrats retweeted his post en masse, calling for federal AI regulation. Two days after Coxon resigned, no less than Barack Obama cited the news and implored fellow Dems to take action on AI by making its dangers central to their campaigning and their Congressional agenda.
Cue messaging like this, from Jon Ossoff on Sunday:
What Ossoff implies there makes some sense: if the U.S. government doesn’t rush inspectors into frontier labs, then there could be an imminent bioterrorism threat that should concern American citizens everywhere. And … Wait, what? Actually, that claim is completely unsupported by the facts we have publicly, and wildly irresponsible to broadcast to millions of people.
As for a global AI treaty that would slow America’s technological progress in partnership with a country that Congress has deemed a hostile foreign adversary, on this week’s episode of Sharp China we talked in depth about why China’s good faith participation in that kind of framework is unrealistic. China would not trust the U.S. side of that deal, probably with good reason. I’ll also add that China has refused to cooperate with the U.S. on nuclear arms control, has cheated on its climate pledges, is already flagrantly ignoring international law with the Philippines in the South China Sea, and has repeatedly sought concessions from the U.S. in exchange for its help stopping the flow of fentanyl into the U.S., only to subsequently withhold cooperation every few years.
In general, it’s been depressing to see how quickly half the government has abdicated its judgment on these questions to industry leaders who have their own agendas, ideologies and blindspots. Real, effective leadership requires listening to experts but not deferring to them, recognizing the world as it is, not how you’d like it to be, and balancing priorities across multiple domains. AI will one day be as elemental to daily life and global industry as electricity or the internet, and in addition to confounding more mundane conversations about liability and negligence on the part of model companies, stoking this panic will permanently taint the technology in the eyes of millions. And to what end? I’m not opposed to tech regulation and have written that we could use more of it, but lobbying for a global AI treaty or an industry-wide slowdown—in an industry that China’s MSS calls “the main battlefield of global scientific and technological competition”—is emotionally satisfying and strategically baffling.
It’s probably smart politics, though. Americans are stressed and confused, and slowing down AI sounds great to a generation of people who still can’t figure out why their refrigerators need a WiFi connection, why new Apple headphones cost $129 and have to be charged, why streaming entertainment is now more expensive than cable ever was, or why every piece of software on earth asks you to pay $9.99/month. Alongside that malaise, and with all due respect to the millions who were waiting their whole lives to see whether anyone could solve the Navier-Stokes problem, AI has not actually improved life for the vast of majority of Americans. Generally speaking, people are less happy than they were before technology transformed the world, society is less unified, and, according to some metrics, everyone is now dumber. To the extent AI’s benefits remain elusive, resistance to further disruption is understandable, and the appeal of “can we just not do this?” arguments will be enduring.
There are also real and obvious risks that will come with AI. Some portion of model behavior remains genuinely mysterious and occasionally surprising even to experts, and some of those surprises are concerning; that’s why testing is important. Also, earlier this year a small team of researchers in California used AI to hack WeChat, an app used by 1.4 billion people. One thing AI is definitely useful for is hacking and spying; security incidents spearheaded by humans, not rogue agents, are likely to proliferate over the next several years as cybersecurity defenses catch up with AI-assisted offensive hacking capabilities. That inevitable spate of hacks may cost billions of dollars in the best case scenarios and could lead to deaths and social upheaval in the worst cases. Meanwhile, I also worry about what this technology will do to education and cognition, how it will be incorporated into workplaces, what it might mean for human intimacy and psychology, the ways in which AI could be abused for surveillance purposes, and yes, whether it will ultimately render all of us unemployable (and what that might do to political stability). All these potential downsides are inchoate in 2026, but government will absolutely have a role to play in helping American society answer some of these questions.
The key point, though, is that those risks can be managed. Problems arise, evidence becomes clear, and we either enforce existing laws or pass new ones; that’s how technology and democracy has always worked. Automobiles lead to more than a million deaths per year, but society has decided that the benefits of automobiles are worth it, and passed laws to mitigate harms. Why should AI be any different? And with respect to the democratic process: what concrete harms has AI yielded to date that would justify the preemptive imposition of a regulatory regime that’s likely to permanently entrench the dominance of market leaders, create a public-private partnership with an oligopoly, and artificially handicap U.S. progress in a competition for global AI leadership with China? If Amodei, Sam Altman, or Elon Musk have evidence to suggest this is urgently necessary, I’d like to see a lot more.
A Technology Lesson
One day when I think about the beginning of the AI era, I’ll remember that AI was all anyone wanted to talk about, and most of the conversations were not particularly interesting. The tools themselves could be genuinely breathtaking, but their precise implications for society were unclear. We responded to that ambiguity with years of sensationalist claims about economic upside (god bless SpaceX’s TAM), fears about catastrophic job loss, dreams about UBI and novel drug discovery and a world without property laws, and of course, a steady drumbeat of existential dread. Not much of this talk was rooted in anything concrete, and freed from the obligation to cite hard evidence, anyone could argue just about anything. AI conversations became an opportunity to tell people about yourself and your biases, if not necessarily AI.
A few weeks ago I wrote that the nationwide data center backlash would peak relatively soon; I expect the anxiety around grave AI risks will be a lot more durable. Governments will have a bias to assert centralized control over this technology and will use alarmism to accomplish that goal, while the AI labs are full of researchers who earnestly believe that AI presents extinction risks. Those employees will soon have billions of post-IPO dollars to donate to organizations that fan the flames of that conversation, and there’s a moral valence to this fight that will be politically useful in Washington even as it’s geopolitically counterproductive. The market-leading labs themselves have limited incentive to downplay any of these concerns, because they are likely to benefit from government regulation that raises the barrier to entry for competitors or forecloses open source competition altogether. When harmful security incidents emerge, driven by humans, the panic will of course intensify.
As to the acceleration of all this rhetoric over the past 10 days, I’ll return to my role as Sharp Tech’s resident normie and admit that when Amodei warned of an AI “botnet” that will take over the internet in the next 12 months, I had to click through to the Wikipedia entry he linked to help me understand the term. Since that post published, I’ve seen a U.S. Senator demand the U.S. government adopt an “emergency footing” to address the threats highlighted by Amodei. The Senate Minority Leader announced on Monday, “Alarm bells are ringing louder than ever, and the Trump administration is looking the other way.” A second Democratic presidential candidate is demanding that Trump pursue an AI treaty that China is unlikely to join and will not comply with if they do. Elsewhere, Andrew Yang appeared on CNBC to claim the internet has already been contaminated by self-replicating agentic code, Joseph Gordon-Levitt (???) has asked the government to shut down all the AI companies, and Netflix has released a new documentary about AI and the end of the world. In case there was any doubt about the impact of all this, at least one poll indicates AI salience has increased dramatically in the past week, and fear for humanity is now widespread among American voters.
For my part, I now know what a botnet is. And I am pretty sure we will be living with some version of this one for at least the next 10 years.
Sharp Text is extension of the Stratechery Plus podcasts Sharp Tech, Greatest of All Talk, and Sharp China. To subscribe and receive weekly posts via email, click here.