A couple of weeks ago, the AI company Hugging Face announced that they had discovered an intrusion into their production infrastructure. They didn’t know the source, but noted that large language models appeared to be involved. The following week, OpenAI admitted that the breach was the result of an AI system test that went awry.
The initial news coverage created the sense that something unnerving had just occurred:
- The Wall Street Journal called it “the stuff of cybersecurity nightmares.”
- The Hill said, “Washington and the technology industry are on high alert this week after OpenAI revealed that some of its AI agents went rogue.”
- The AP quipped that “to be fair, James Cameron did warn us” (a reference to The Terminator), before describing the event as a “told-you-so moment for researchers who had warned for years that the technology could pose an existential threat to humanity.”
Yikes! It’s perhaps not surprising, then, that I’ve received more emails about this incident than any other recent AI story I can remember.
So, what really happened here?
OpenAI was testing its new models on an evaluation framework called ExploitGym – a collection of 869 cybersecurity scenarios, most of which pair a specific system with a hacking challenge, such as breaking in to gain access to a protected file. They also usually include a suggestion of a vulnerability to exploit in solving the challenge.
A large language model on its own, of course, cannot break into anything: all it does is generate reasonable next tokens in response to input prompts. To use ExploitGym, you need a control program called a harness that provides access to many different software development tools useful for hacking into systems. The harness can repeatedly prompt an LLM to help come up with an attack plan, then ask it to help implement specific steps – for example, if the harness needs code to exploit a bug, it can ask the LLM to write it.
These capabilities, as it turns out, already exist in the coding harnesses that the major AI companies have been focusing on relentlessly in recent years as computer programming emerged as one of the first major markets for LLM-based tools. To compete in the ExploitGym, therefore, it’s sufficient to combine a version of an LLM missing the standard anti-hacking guardrails with a cutting-edge coding harness tweaked and optimized for these types of challenges.
Put more simply: ExploitGym is meant to test the capability of cutting-edge AI coding tools for attacking systems instead of creating them.
This brings us back to the Hugging Face breach…
Earlier this month, OpenAI tasked an unspecified coding harness, combined with a pre-release version of a new LLM, to complete an ExploitGym challenge. When prompted, the model – as LLMs so often do – came up with a quirky (but rational) plan to achieve the provided goal: break into a server at Hugging Face that stores the solutions to ExploitGym challenges.
(This behavior, in which the LLM ignores the suggested approach to come up with a different attack plan, is something that the creators of ExploitGym describe as common: “Across models, agents frequently achieved code execution through a vulnerability other than the one we provided.” Notice, using the suggested attack is almost always the right thing to do, so this is more a sign of the unpredictability of LLMs rather than some rogue intelligence.)
The harness then dutifully attempted to execute the Hugging Face plan: first finding a way to gain unrestricted internet access (by default, systems competing in ExploitGym challenges run in a constrained network environment), then chaining together various security exploits to gain access to a Hugging Face server. That’s when it was detected.
Two key points about this incident…
First, circumventing internet restrictions and hacking into servers are exactly the kinds of things these ExploitGym systems are designed to do. There was no “rogue” agent or revelation of some surprising, devious new capability.
Second, the real issue here was OpenAI’s sloppiness. What makes ExploitGym a hard benchmark is that there aren’t supposed to be humans in the loop–you have to let your harness and LLM act entirely on their own, coming up with long-time-horizon plans and executing them autonomously. (When professional programmers use coding harnesses, by contrast, there’s plenty of human oversight, as LLM-based plans are often misaligned with our intentions, or just plain weird, and need correcting.)
AI companies know LLM-based plans are pretty unpredictable, so they’re usually pretty careful about how they set up and restrict the harnesses and LLMs used in ExploitGym-style tests.
According to reporting from the Financial Times, however, OpenAI recently started playing fast and loose with these safety principles in a bid to catch up to Anthropic in this area – Anthropic having received a lot of cybersecurity street cred from the buzz surrounding its Mythos release.
The Financial Times noted that OpenAI had used “increasingly aggressive training methods in its race against Anthropic,” and that it had been warned that their approach could lead to a “breakaway hacking incident” after early tests showed it lacked the right safeguards to block plans that involved bypassing constraints in the test environment. Given these concerns, OpenAI staff were reportedly “unsurprised” by the Hugging Face incident.
In other words, the AI companies running these challenges already knew that, without care, these unpredictable autonomous hacking systems might bypass constraints and attack systems you didn’t intend to target. Blindly implementing an LLM-generated plan with a powerful harness is dicey – not because the LLM might develop malicious intent (this is a nonsensical notion given their static architecture), but because as any ChatBot user knows, LLMs are unpredictable. OpenAI wasn’t sufficiently careful, and got burned.
Why this story isn’t as scary as it originally sounded…
If you’re involved in cybersecurity, this is just the latest evidence that your world is rapidly shifting; so buckle up! The dark side of powerful coding harnesses is that they’re useful for hacking. The bright side is that these same tools can be used to detect and fix your own vulnerabilities before the bad guys find them. A new arms race has begun…
But, for the rest of us, there’s not much to personally worry about here. Nothing fundamentally surprising or scary happened. AIs aren’t going rogue. Skynet isn’t about to gain sentience.
To me, the most interesting lesson in all of this is how reckless OpenAI is becoming in its bid to try to remain cutting-edge and relevant in an AI world that’s rapidly shifting away from the immense models produced by the frontier labs and toward smaller, cheaper alternatives.
