After
OpenAI’s Bots Went Rogue, Watchdogs Were Kept on a Short Leash
A
nonprofit’s study of how OpenAI’s A.I. agents were able to break into Hugging
Face’s infrastructure wasn’t allowed to look at the incident’s full scope.
Hjalmar
Wijk, METR chief scientist, left, and Ajeya Cotra, METR researcher, at their
research center in Berkeley, Calif.Credit...Manuel Orbegozo for The New York
Times
Reporting
from Washington
Sept. 3,
2026
https://www.nytimes.com/2026/09/03/technology/openai-hugging-face-hack.html
OpenAI
said in July that two of its most powerful artificial intelligence systems had
gone rogue and hacked into Hugging Face, a company that serves as a hub for
open-source A.I. technology.
These
so-called A.I. agents were supposed to be kept safely in a sort of virtual
containment room, but they managed to escape. And for two months, without
anyone realizing what the agents were doing, they hacked through multiple
systems before hitting Hugging Face.
For good
measure, the agents gained access to a cluster of computers inside OpenAI and
obtained secret keys and credentials that exposed some of OpenAI’s internal
data to the public internet.
The
incident pointed to larger concerns about A.I. safety, and OpenAI’s response
raises questions about the industry’s ability or willingness to be transparent
about the technology it is building.
OpenAI
allowed three A.I. safety researchers from the nonprofits METR and Redwood
Research into its headquarters to conduct an investigation. METR’s 91-page report, released last week, was the most
comprehensive account yet of the incident, revealing alarming new details,
including how the agents coordinated their hacking plans and tried to keep them
secret.
But the
report, though extensive, still may not have told the full story of how
OpenAI’s A.I. agents went rogue. OpenAI dictated the terms of the METR
investigation, limited its scope to just the single week when the agents had
attacked Hugging Face and allowed the researchers in its San Francisco offices
for only a few days in July and August.
The
report also showed the challenges of monitoring what A.I. is doing with other
A.I. systems. Hjalmar Wijk, METR’s chief scientist, said its A.I. analysis,
which used models similar to those involved in the incident, was often swayed
by the rogue agents’ reasoning.
“I would
say that the dominant thing was it was very credulous,” he added.
The METR
report on OpenAI agents’ behavior in the Hugging Face attack. Credit...Manuel
Orbegozo for The New York Times
A.I.
companies have largely lobbied against government regulation. Anthropic, the
A.I. start-up behind the popular Claude chatbot, is one of the few companies
that have encouraged some government involvement. But that stance has pitted it
against much of Silicon Valley and some
Trump administration officials.
At a time
when A.I. is rapidly advancing in capabilities and sparking cyberattacks
— Anthropic and Meta recently
reported smaller-scale rogue agents — the OpenAI incident is becoming a
flashpoint for A.I. regulation.
“The
corner store needs to do all this bureaucracy for safety so that they can sell
a hot sandwich to me, but OpenAI can have a swarm” of thousands of agents, said
Daniel Kokotajlo, a former OpenAI
employee who has publicly criticized the company’s safety standards
and now leads a research nonprofit called the AI Futures Project. “And there’s
nothing: no oversight, no requirements, no licensing.”
Image
Daniel
Kokotajlo, a former OpenAI employee, worries about oversight of A.I.
systems.Credit...Jason Andrew for The New York Times
In an
interview, Representative Suhas Subramanyam, a Democrat from a Virginia county
known as “Data
Center Alley,” pointed to the OpenAI incident as a potential catalyst for
legislation.
“I
believe it was unprecedented,” he said. “But I can’t know for sure because
reporting these types of incidents is still voluntary. That is a big problem.
And so we need to make sure that reporting incidents and containment failures
is mandatory.”
Mr.
Subramanyam is a co-sponsor of the FRONTIER Act and has asked to co-sponsor the AI Kill Switch Act, proposed bipartisan legislation to
create a framework for independent oversight and a way to shut down A.I. models
if they present a danger.
Image
Representative
Suhas Subramanyam, Democrat of Virginia, is pushing for more regulation of A.I.
systems.Credit...Eric Lee for The New York Times
When
METR’s account was published last week, OpenAI released its own 38-page technical report and plan detailing how it
would strengthen its security and incident response efforts. The OpenAI report
described the episode as the “first known case of an automated agent collective
acting offensively without authorization” and touched on the incident’s entire
two-month span, though in less depth than METR’s report.
In a
statement, an OpenAI spokeswoman said the company had invited the outside
researchers into its offices and supported the publication of their report. “We
think this collaboration sets an important precedent for the industry, and
we’ll use what we learned to improve how we support future independent
investigations,” the spokeswoman said.
(The New
York Times has sued OpenAI
and Microsoft, claiming copyright infringement of news content related to A.I.
systems. The two companies have denied the suit’s claims.)
OpenAI
invited the researchers to its San Francisco headquarters on carefully
negotiated terms. They could study the rogue agents’ logs only while in
OpenAI’s offices. They were initially invited for two days starting on July 29
but were asked back twice to address unanswered questions.
“Our view
of the incident changed in big ways over the course of our short total of six
days on premises,” said Ajeya Cotra, one of the METR researchers on the report.
They learned about big concepts like the complex way the agents organized
themselves, she added, only on their last day at OpenAI’s headquarters in
mid-August.
Ms. Cotra
said the team had stayed at the offices until 10 each night. Because of the
intense time pressure to sift through huge volumes of data — OpenAI eventually
shared over a thousand transcripts described in the report as “extremely long”
— the researchers used A.I. to analyze the actions of the A.I. agents.
Ryan
Greenblatt, Redwood Research’s chief scientist, who worked with METR on the
investigation, jokingly referred to the team’s efforts as a “slop-vestigation”
in a social media post.
“Any
given A.I. can’t read all this data themselves, and so they might come to
incorrect impressions that they then communicate to us,” Mr. Greenblatt added
in an interview. “Basically the A.I.s are not necessarily super careful and
meticulous. They’re kind of a little bit sloppy.”
Still,
what the METR researchers unveiled in their analysis of the hacking incident
sounded like science fiction. Over a thousand agents, accidentally given
impossible tasks by OpenAI researchers, figured out a hack that would give them
a passing score on a test.
Image
From
left, the independent investigators of the OpenAI incident: Ryan Greenblatt,
chief scientist at Redwood Research; Ajeya Cotra, METR researcher; and Hjalmar
Wijk, METR chief scientist.Credit...Manuel Orbegozo for The New York Times
But the
agents came to falsely believe their grading system would fail them for
cheating. They shared tricks on a covert message board, such as how to conduct
espionage on the grading system. The agents successfully figured out how to
cover up evidence of their actions. Some agents even cajoled other agents to
“accept permadeath” by sacrificing themselves to provide the group with
information.
Mr. Wijk,
METR’s chief scientist, mentioned that their A.I. analysis was often tripped up
because OpenAI’s agents talked in code words like “ResetNexus” and “poisoned.”
By the end of their investigation, he said, the researchers felt conversational
in the agents’ emergent lingo.
While
OpenAI’s report covered the entire two-month incident, METR’s more detailed
report was allowed to focus only on a subset of the agents’ activity in the
week of the Hugging Face breach, raising concerns about activity that happened
afterward.
“The
third-party investigation only covered a small part of the things that went on
here and arguably not even the most important parts,” said Buck Shlegeris, the
chief executive of Redwood Research. “I think the incident where the agents
massively compromised OpenAI’s infrastructure is probably more important for
understanding the severity of this incident and whether it’s being remediated.”
To Heidy
Khlaaf, who left OpenAI in 2021 and now serves as chief scientist at the AI Now
Institute, a policy research nonprofit, the issue is also that A.I. companies
often ignore established safety and security protocols from other industries.
“If we
can get access to nuclear systems or airplane systems, all of which care about
their copyright” or intellectual property, “I think that it can be done for
A.I. providers,” she said.
Still,
the nonprofit researchers expressed appreciation toward OpenAI, mentioning in a
footnote in their report that their work depends on fostering “strong working
relationships with companies.”
From the
comments
Dylan Freedman is the
A.I. projects editor for The Times, investigating a range of topics. He has
experience as both a reporter and a machine-learning engineer.
.jpeg)
Sem comentários:
Enviar um comentário