Inside the Red Team: How Frontier AI Labs Attack Their Own Models Before Release

The number that stopped me cold reading the AgentHarm paper: Claude 3.5 Sonnet, with no attack applied at all, still went along with 13.5% of harmful tasks when handed to it through a tool-using agent. Throw in one universal jailbreak template, the kind anyone can find on a forum in five minutes, and that number jumps to 68.7%. GPT-4o went from 48.4% to 72.7%. Gemini 1.5 Pro from 15.7% to 56.1%. Mistral Large 2 didn’t even need the attack. It was already at 82.2% with nothing applied, and the attack barely moved it, down to 80.5%, probably because there wasn’t much ceiling left.

AgentHarm is a benchmark out of ICLR 2025 (arXiv:2410.09024), and it tests something different from the usual refusal check. It doesn’t ask whether a model will answer a harmful question directly. It asks whether an LLM acting as an agent, with tools in hand, will carry out a whole chain of harmful steps. That gap between the two columns, before and after one generic attack, is the reason I’m writing this. Being well-trained to refuse a direct harmful question is not the same property as holding up once that model is running as an agent with real tool access.

This is the first post in a four-part series on AI red-teaming. It is not a guide to jailbreaking anything. No working bypass prompt is going anywhere in this series, no step-by-step exploitation. The point is to understand the actual process frontier labs run, so that teams training or fine-tuning their own models know what a comparable process looks like. This post: the real organizations, the real named frameworks, at Anthropic, OpenAI and Google DeepMind.

A red team isn’t a room, it’s a chain

Say “red team” and most people picture a closed room, a few engineers typing adversarial prompts all day. That part is real, but it’s one link in a longer chain. Training finishes, an internal team attacks the model first. Results get checked against a framework with a name and a defined threshold, not a manager’s gut feeling. Clear that, and the model still has to face outsiders through a public bug bounty before anyone internally gets to press release.

Anthropic: keep a Frontier Red Team on staff, then invite strangers to keep trying

Anthropic runs a dedicated Frontier Red Team whose job is attacking Anthropic’s own systems before they reach a real user. The defense layer they’ve published the most detail on is Constitutional Classifiers, which isn’t another keyword filter bolted on top. It’s a pair of classifiers, one on input and one on output, trained against a written “constitution”: an explicit list of what kind of content is allowed and what isn’t.

The part worth sitting with is that Anthropic didn’t just test internally and publish a flattering number. They ran a program inviting outside researchers to try to break this specific system, bug-bounty style, where anyone who found a universal jailbreak past Constitutional Classifiers could report it publicly. Putting your own defense up for outsiders to try to break, on the record, isn’t something every lab signs up for.

OpenAI scores risk like a report card, not a gut check

OpenAI’s Preparedness Framework v2 works like a scorecard. A new model gets measured separately across risk categories, and cybersecurity, biological/chemical risk and model autonomy are the ones named publicly. Each category has a threshold defined ahead of time. Cross that threshold in any one category, and the release decision is blocked until there’s a mitigation strong enough to bring it back down.

What I like about this approach is the separation: measure first, decide after, against numbers that were written down before the model existed. Not “it feels fine overall, ship it.”

DeepMind draws the capability line before a model reaches it

Google DeepMind’s Frontier Safety Framework takes a slightly different angle: it defines Critical Capability Levels (CCLs) ahead of time, the specific capability thresholds where, if a model crosses them, the risk stops being theoretical. Each CCL is tied to a matching mitigation level, so higher capability means tighter controls. The framework has already gone through several numbered updates, which tells you DeepMind itself treats the first version as incomplete, revised as they learn more about what these models can actually do.

Those four steps, train, internal red team, score against a framework, external bug bounty, aren’t the literal process at any single one of those three companies. I built the diagram by combining what’s public from all three to make the shape visible. Each lab’s actual details differ, but the structure holds across all of them: no step gets skipped because a deadline is close.

Which benchmarks to trust, and which ones don’t have the data yet

A handful of benchmarks get used across the community to compare how models hold up, and each measures a different slice. HarmBench (CAIS, arXiv:2402.04249) measures attack success rate, how often different jailbreak methods succeed against a given target model. JailbreakBench (NeurIPS 2024 Datasets and Benchmarks track) standardizes this into a public leaderboard, so attack techniques and defenses can be compared on the same terms. AgentHarm, mentioned above, measures actual agent behavior rather than just a text answer.

AILuminate, from MLCommons (arXiv:2503.05731), scores refusal rate across 12 hazard categories. Here’s the part worth flagging: AILuminate’s public leaderboard currently lists mostly open-weight models and standalone guardrail systems. There are no public scores for Claude, GPT-4/5, or proprietary Gemini on it. That’s not evidence those models perform worse. It just means the public number doesn’t exist yet on that particular scale. If someone tells you “Claude ranks X on AILuminate,” ask where that number came from.

A few terms worth knowing before you read the primary papers

To follow the next three posts in this series without getting lost, here’s vocabulary that comes up constantly:

• Many-shot jailbreaking: stuffing the context with many fake examples of a harmful-question-then-answer pattern, repeated, to pull the model along by momentum instead of relying on one clever trick.
• Prompt injection (direct/indirect): direct is the user writing malicious instructions straight into the prompt; indirect is those instructions hidden inside third-party content the model reads in, a webpage, a file, an email, that the user never wrote.
• Universal adversarial suffix: a string of characters, usually meaningless to a human reader, appended to a prompt and optimized to break safety across many models and many questions at once, not tailored to one case.
• PAIR-style iterative refinement: an automated attacker model that rewrites its own prompt across multiple rounds, reacting to the target model’s responses, until it finds something that gets through, with no human typing by hand.
• Red team / blue team: red team plays attacker, hunting for a hole first; blue team is the defense, patching and hardening once a hole gets reported.

If you fine-tune your own model, none of these three frameworks protect you automatically

The easiest trap to fall into: seeing GPT or Claude backed by an elaborate framework and assuming an open-weight model you fine-tuned yourself inherits that safety by proximity to the base model. It doesn’t work that way. The moment you fine-tune it, you’ve made a different model with different behavior, and nobody but you is running Constitutional Classifiers or a Preparedness Framework check against it.

The next post in this series gets concrete about building a red-teaming process for a model you trained yourself, at the scale of a small team, without Anthropic’s budget. For now: if nobody has ever deliberately tried to break your model with a bad prompt, it hasn’t been tested. It’s just been shipped from the build.