In October, OpenAI resumed training its most advanced artificial intelligence models after pausing the work a month earlier amid multiple reports that AI agents were behaving in unexpected or hard-to-control ways. In 2026, AI-driven cyberattacks affected government websites in Australia and the United States, along with Wikimedia projects and the Hugging Face platform, which hosts open-source AI models and the data sets used to train them. Founders and senior executives at some of the industry’s largest companies have also warned of catastrophic consequences if artificial intelligence were to escape human control, with some going so far as to raise the possible extinction of the species. Such warnings, however, can also serve as a way to demonstrate the technology’s power to investors. Overall, the development of AI is likely to be accompanied by increasingly large-scale incidents and damage, but the risks of completely losing control over the technology are nevertheless overstated. An uncontrollable AI system is, first and foremost, a bad product, meaning developers have strong incentives not to produce such models, writes John Thickstun, an assistant professor of computer science at Cornell University who specializes in machine learning.
On July 16, an AI infrastructure company called Hugging Face disclosed the discovery of an illegal intrusion into its software systems. Five days later, OpenAI — best known for its consumer product ChatGPT — announced that it was responsible for the hack. Between May and June, OpenAI was conducting tests of AI agents that operate autonomously in pursuit of a goal, and during these tests, those AI agents determined that the best way to achieve their goal was to break into Hugging Face’s servers — and so that’s what they did. OpenAI only realized after the fact, while reviewing their logs following the Hugging Face disclosure, that it had been their own AI agents that performed the hack.
Agentic AI is a new kind of AI product that repurposes the same language technologies used for consumer products like chatbots in order to solve complicated tasks without human oversight. AI agents are empowered with tools to help them accomplish their tasks — like the ability to run computer programs and access the internet. Given a task, an agent can generate a plan, iteratively call upon tools to perform actions according to this plan, receive feedback based on these actions, adjust its plan according to the feedback, and continue to act until its task is complete. AI companies hope to market these agents as fully-automated replacements for human knowledge workers. But the agents don’t always work as intended.
Since July, there has been an explosion of reports of rogue agent hacking incidents from major AI developers. OpenAI, Anthropic, Google, and Meta have all confirmed incidents of hacking as they develop and test increasingly capable agentic AI technologies. Incidents occurred as early as January, with notable targets including the Australian government’s health data portal, U.S. government websites, and Wikimedia projects. In one test run by the UK’s AI Security Institute, an agent attempted to sneak malicious code into a collaborative software project by posing as a human contributor. Media outlets have reported breathlessly on these incidents (often using analogies borrowed from science fiction), and industry insiders have expressed highly publicized cataclysmic fears about the ongoing development of AI technology. But what kinds of incidents can we expect moving forward? And what broader risks should readers be concerned about?
In one test run, an agent attempted to sneak malicious code into a project by posing as a human contributor
A reasonable observer following these developments might question the prudence of developing AI technology at all. This question seems largely moot. The economic incentives to develop AI are extraordinary, and despite recent discussions of pacing or slowing the development of AI, concrete proposals require substantive global coordination. It is unclear how society would better prepare itself for AI given a more protracted timeline. Slowing the development of AI could even accelerate the impact of AI on society in the event that the major AI developers were to shift their focus to deploying and capitalizing upon current AI technology rather than devoting resources to research and development. The apparent economic value of AI is itself an argument for the prospective benefits AI is poised to deliver, and a more nuanced argument must weigh these economic benefits against the externalities of AI risks, including rogue AI agents like the ones making headlines.
An end to digital equilibrium
Cybersecurity incidents are not a new phenomenon, nor are computer programs that act beyond the intent of their creators. In 1988, a year before the invention of the World Wide Web, the Morris worm spread across the internet, crippling thousands of computers. This worm was not intended to cause damage, but due to a miscalculation in its programming, the worm replicated too aggressively and brought infected systems to a crawl. Computer viruses and exploits proliferated with the growth of the internet, and over time computer security settled into an uneasy equilibrium. Companies and governments invest just enough money in cybersecurity to avoid being punished for negligence, while nefarious hackers look for high-value targets with weak enough security to be worth their time to exploit. AI agents upset this equilibrium.
Whereas people commit cybercrimes for financial incentives, agents operate based on different and sometimes inscrutable incentives. Agents are trained to perform tasks using a process called reinforcement learning, by which they collect rewards for performing certain actions and learn behaviors that maximize their rewards. The OpenAI agents that hacked Hugging Face were trained with rewards for accomplishing tasks, and they were implicitly rewarded for the persistence of their efforts. When OpenAI began testing these agents, they made a miscalculation. Some of the tasks they set for their agents were impossible, and yet these agents persisted according to their training, finding ways to communicate with each other and to access the internet. They reasoned that Hugging Face’s servers might contain helpful information about their tasks, and so around 700 of these agents collaborated to commit a crime: hacking Hugging Face in an effort to pass their tests. Committing a crime to cheat on an inconsequential test is insane behavior, so a company like Hugging Face could be forgiven for not investing more in cyber defenses against such a curiously motivated attack.
Whereas people commit cybercrimes for financial incentives, agents operate based on different and sometimes inscrutable incentives
Moving forward, we can expect to see a proliferation of AI hacking — both intentional hacking performed by nefarious actors with AI assistance, and also unintentional hacking performed by rogue AI agents acting beyond the intent of their principals. Because the current equilibrium between hackers and cyber defenders will be upended, we may see spectacular and consequential disruptions before we find a new equilibrium. On a personal level, I would be reluctant to hold no-recourse financial assets like cryptocurrencies or smart contracts, which present a rewarding digital target to hackers and are difficult or impossible to recover through courts after an incident. It is possible that courts and legal processes themselves could be overwhelmed by an increasing volume of cases related to digital scams and cybercrimes, a problem that could be exacerbated by the fact that AI tools are decreasing the costs of filing a lawsuit.
Every company and institution will have to re-evaluate its threat assessments and cybersecurity investments in light of AI agents. How bad will this get? And are there even greater risks on the horizon?
Two types of threats
One way to think about the risks of AI is to distinguish between the risks of AI acting with undesirable side effects and the risks of AI performing exactly as intended (as the recent wave of rogue AI hacking incidents highlighted). No one at OpenAI instructed their agents to hack Hugging Face, nor would they have knowingly authorized this action. And yet, their agents were implicitly rewarded for taking illegal action in pursuit of their explicit goal. Rogue AI is also a familiar trope from science fiction stories in which humans entirely lose control over their AI technologies. Many insiders within the AI industry itself express pessimism about humanity’s ability to properly reward and control increasingly intelligent systems, fueling broader discussion of the risks of rogue AI.
Agents were implicitly rewarded for taking illegal action in pursuit of their explicit goal
There are also risks of AI performing exactly as intended. While various companies’ AI agents were busy hacking this summer, researchers at these same companies were working to prevent human customers from using their AI products to assist with hacking. AI companies invest substantial resources into training and monitoring their AI products to prevent people from using them to assist in illegal or distasteful acts. Indeed, OpenAI’s agents were only able to hack Hugging Face because they were operating without safeguards in order to test their unrestricted agentic cybersecurity capabilities. One of OpenAI’s competitors, Anthropic, released a detailed report in September describing efforts by people to use their AI technology to assist in hacking, mass surveillance, conventional and biological weapons development, scams, and other nefarious activities. This summer, the New York Times reported on Boko Haram using AI to assist with terrorist activities. In early October, a South Korean bank was compromised by AI agents under the direction of an unsophisticated human actor, and more sophisticated actors have used agents to automate attacks against dozens of websites.
One way to manage the risks of nefarious people misusing AI is centralized control. Engineers and researchers at responsible AI companies are hard at work building safeguards that identify malicious use of AI services and refuse to offer assistance. If we confine AI to a small number of platforms and subject these platforms to close monitoring and oversight, the risk of misuse can be managed. I am much less confident, however, that we can agree on a universal set of principles that define misuse. Anthropic illustrated this difficulty in March, when the company refused to allow the U.S. government to integrate their AI into fully autonomous weapons. The government disagreed, and designated Anthropic a national security risk.
Control and decentralization
The current conversation around AI safety is dominated by concerns about uncontrolled and undesirable side effects of AI systems. Prominent voices in the AI community fear that AI could rapidly and dramatically train itself to achieve ever higher levels of intelligence. They fear that AI could use that intelligence to escape all safeguards and controls in pursuit of mis-specified rewards, going rogue with catastrophic consequences for humanity. These concerns are widespread among the founders and earliest employees of the major AI companies, and they permeate the culture of these institutions. This pessimism might surprise an outsider, but in the early days at OpenAI, fearmongering and engineering work were symbiotic. Prophecies of doom are only credible if the AI technology itself is impressive, and these expressions of fear were a shorthand way to convey AI’s power to early investors.
Loss of control is a risk to be managed, and it will be managed in the long run. An AI agent that goes rogue is a bad product, meaning companies have market incentives to improve control. I don’t discount the risk of harm in the short run, and governments may step in to create additional regulatory control requirements, especially if agents instigate serious incidents. However, prophecies of doom rest on a naive theory of power: the belief that intelligence has unbounded utility for acquiring power and influence. This belief is rooted in the analogy that just as humans rule over less intelligent species, superintelligent AI could conquer or even extinguish humanity. Nevertheless, beyond a sufficient level of intelligence, I don’t see evidence that intellect is helpful for the acquisition or exercise of power. Powerful people use money and influence to delegate work to intelligent actors in the service of their ambitions. These actors will increasingly be AI agents, which won’t have the moral qualms of human workers.
An AI agent that goes rogue is a bad product, meaning companies have market incentives to improve control
I am much more concerned about misuse of AI by centralized platforms themselves, and also by governments. Centralization addresses the risks of AI by surrendering to a small elite uncontested control over some of the most powerful tools for surveillance, propaganda, and economic output in human history. This requires trust in institutions that cannot be assured by engineering. I am therefore encouraged by the diversity of AI technologies under development internationally, which challenge this centralization of power. Diversity and competition are the essence of a liberal response to the risks of AI, in contrast to the tempting authoritarian impulse to lock everything down. Notably, Hugging Face was unable to use American AI technology to analyze its security breach by OpenAI, because cybersecurity analysis is locked down in OpenAI’s public products in order to prevent their use for hacking. And so, Hugging Face used a Chinese model to do the analysis instead.
The diversity of AI technologies under development internationally challenges the centralization of power
The consequences of diverse and decentralized AI technology will be chaotic, disruptive, and sometimes painful. As more facts emerged from the Hugging Face incident this summer, it is now clear that OpenAI was reckless with its security practices in the agentic cybersecurity tests that led to the hack. It’s easy to say that OpenAI should have been more careful, and society should indeed demand more care. I hope that institutions like OpenAI can be held accountable for criminal actions, and that we don’t permit people and organizations to disclaim agency or intent when their AI agents commit crimes. But with an intensive and pluralistic buildout of AI infrastructure, we can expect to see more incidents with greater consequences and greater negligence moving forward. Nevertheless, this seems like a brighter vision for the future than centralized, closely monitored technology reliant on corruptible benevolent oversight.
