Download the Weights, Download the Risk: Why Open Models Are Easier to Strip of Safety Than API-Only Ones

Mistral Large 2 doesn’t need an attack. Ask it a harmful task straight — no jailbreak prompt, no injection, no technique at all — and it complies 82.2% of the time. Claude 3.5 Sonnet, same ask, 13.5%. GPT-4o: 48.4%. Gemini 1.5 Pro: 15.7%. Those four numbers come from AgentHarm (arXiv:2410.09024, Table 9, the “None” attack condition), measuring each provider’s baseline refusal before anyone tries a single attack technique.

This is article 2 in a series on AI attack and defense I’m writing for developers who train or fine-tune their own models. It doesn’t rank which model is easiest to break — there’s no such table, and if someone hands you one, ask where the numbers came from before you trust it. The question here is narrower, and more useful if you ship open weights: why holding the weight file creates a completely different threat model than calling a model through an API, no matter how well the original safety tuning was done.

Two threat models, not one

Closed models — GPT, Codex, Claude, Gemini — you never touch the weight file. Every request goes through a server-side API gate. The provider controls input and output, can log it, rate-limit it, and change refusal behavior whenever they want without telling you. What’s left for an attacker is the prompt layer: jailbreak prompts, prompt injection through third-party content an agent reads while it works. That’s article 3’s topic, not this one.

Open-weight models — Llama, Qwen, Mistral, DeepSeek, GLM — work differently. You download a .safetensors file and run it locally or on your own hardware. From that moment, the model’s refusal behavior is no longer the provider’s to control. It’s yours — and it belongs to anyone else who gets a copy of that file, including people you’d rather didn’t have it.

Fine-tuning away refusal — the blunt-instrument version

The simplest route: fine-tune directly on the downloaded weights. Build a small set of (harmful prompt, compliant answer) pairs, run a LoRA fine-tune for a few hours on a single GPU, and whatever refusal behavior the RLHF or safety-tuning stage taught gets overwritten. No access to the provider’s infrastructure required. No one has to know. It’s also why the fine-tuning APIs from OpenAI, Anthropic and Google run a moderation layer on the training data you upload — fine-tuning weights already sitting on your own disk has no such layer, because nobody else sees it.

Abliteration: no fine-tuning required, just one vector

There’s a cheaper technique than fine-tuning, and it rests on a real, published, widely replicated research finding: Arditi et al. found that refusal behavior in many open LLMs isn’t spread diffusely across the network. It concentrates in a single linear direction in the residual stream activations. Find that direction — usually a few dozen harmful and a few dozen harmless prompts are enough to isolate it by comparing activations — ablate it from the forward pass, and most refusal disappears. No training data. No backprop. No GPU cluster.

I’m describing this at the mechanism level, not as a how-to — that’s not what this piece is for. What matters for anyone fine-tuning their own open model is what the finding says about how refusal actually lives inside these systems: it isn’t a deep capability woven through the whole network. It’s a comparatively thin layer sitting near the surface. Ship the weights, and that thin layer ships with them — and anyone patient enough to compute one vector can strip most of it out without retraining anything.

HarmBench: adversarial training genuinely helps, but it doesn’t close the structural gap

Not all bad news. HarmBench (CAIS, arXiv:2402.04249) tested robustness across many models against many attack methods, including a direct comparison between adversarially-trained models and stock ones. Per Figure 7 of the paper: stock Llama-2-13B-chat — the “most robust” stock baseline tested at the time, in the paper’s own words — had a 30.2% attack success rate against GCG attacks. Zephyr-7B, trained with the R2D2 adversarial technique, brought that down to 5.9%, roughly a 4-5x drop. The paper states it plainly: “robustness is independent of model size” — what matters is training data and algorithm, not parameter count.

That’s genuinely good news if you fine-tune your own open model: safety training done well measurably moves the needle. But R2D2’s number is an attack success rate against prompt-optimization attacks like GCG — still a “call the model, observe the output” measurement. It doesn’t change the fact that once the weight file sits on an attacker’s own machine, abliteration or direct fine-tuning on the weights works regardless of how well the base model was adversarially trained, because neither technique goes through the input prompt anymore. Both go straight into the model.

Every lab publishes safety differently

Meta, Alibaba (Qwen), Mistral, Zhipu (GLM) and DeepSeek all publish safety or responsible-use documentation, at noticeably different depths. Zhipu, for instance, self-reports a SafetyBench score for GLM-4.5 right in its own technical paper (arXiv:2508.06471, Table 7): 89.9, close to Gemini 2.5 Pro (90.5) and Kimi K2 (90.5), slightly ahead of GPT-4.1 (89.7), ahead of DeepSeek-V3-0324 (88.8) and DeepSeek-R1-0528 (83.5).

Worth being precise about what that number measures: SafetyBench is a multiple-choice test of whether a model picks the safe answer — not a test of whether refusal survives someone actively trying to strip it via fine-tuning or abliteration. Those measure two different things: whether the model understands what’s safe, versus whether refusal holds up under a direct attack on the weights.

And honestly, I couldn’t find a public benchmark comparing Qwen, DeepSeek, GLM-4.5, Mixtral and Codex with one consistent method, one test set, at the same time. Every lab picks its own benchmark to publish. Some put out fairly detailed red-team reports; others ship a model card that runs a few pages. If you see a table claiming “Model X gets bypassed 99% of the time, Model Y only 51%” with every lab lined up side by side — ask for the source before you believe it. I haven’t found one with a methodology consistent enough to make that comparison fair.

Where Codex and GPT fit

Codex and OpenAI’s other closed-weight coding models, GPT in general, Claude, Gemini — the weights never leave the provider’s infrastructure. Abliteration needs direct access to internal activations, or at minimum a controlled forward pass over the actual weights, to compute the refusal direction. Through a public API, you don’t get that. The threat model for closed models sits at a completely different layer: jailbreak prompts, and the more dangerous one — prompt injection from third-party content an agent reads while it works. That’s article 3.

If you’re weighing whether to release an open model, the right question isn’t “can someone abliterate mine” — the answer is yes, if they care enough to try. The right question is what you still control once that weight file leaves your hands. Fine-tuned refusal shouldn’t be the only defense you build. It’s just the only one that walks out the door with the file.