Here’s a number worth sitting with: raw Claude Opus 5, given the ARC-AGI-3 benchmark directly, scores roughly 30% on the RHAE metric. Wrap the exact same model in NVIDIA’s new AVO harness — no fine-tuning, no different weights — and it scores 100.00 across 183 levels spanning 25 environments. Same model. Same weights. Same API. The difference between “one in three” and “every single one” was entirely in the scaffolding around the model, not in the model itself.

I’ve been saying some version of “the harness matters more than the model” in almost every architecture post I’ve written this year, and AVO is the cleanest empirical proof of that thesis I’ve seen so far — not because it’s a clever demo, but because NVIDIA published the ablation. That’s the part worth digging into.

What AVO actually is

AVO stands for “Agentic Variation Operators,” and despite the fancy name, the architecture is something most of us building production agent systems will recognize immediately, just executed with more discipline than most teams bother with:

  • Persistent memory across the whole task, not just within a single context window — the agent doesn’t re-derive what it already learned about an environment’s rules every time it hits a new level.
  • A supervision layer that watches for stalled search trajectories and redirects them, instead of letting the agent grind on a dead-end approach until it runs out of budget.
  • Environment-specific tool integration — the harness doesn’t hand the model a generic toolset and hope; it wires up tools scoped to what each ARC-AGI-3 environment actually needs.
  • An explicit inspect → plan → implement → evaluate loop, run as discrete phases rather than letting the model free-associate its way through a task in one long completion.

None of these ideas are new individually. What’s notable is that NVIDIA ran the controlled comparison — same base model, harness on vs. off — and published both numbers instead of just leading with the headline 100%. That’s the difference between a marketing claim and a real result you can reason about.

The efficiency number is the one I actually care about

The 100% score gets the headline, but the number that matters more for anyone running agents in production is this one: AVO was 12% more action-efficient than VISTA, the prior leaderboard leader, while achieving a higher score. Action efficiency is a proxy for cost — every wasted action is API spend and wall-clock time you didn’t need to burn. A harness that gets you to the right answer in fewer steps isn’t just “cleaner,” it’s cheaper to run at scale, and that’s the metric that actually determines whether an architecture is viable for a real workload versus a benchmark run you do once.

There’s a second data point in the same release that reinforces this: in a separate GPU-kernel optimization task, AVO ran a 7-day autonomous search and shipped kernels up to 10.5% faster than FlashAttention-4 on a DGX B200. That’s not a toy benchmark — that’s the harness architecture generalizing to a domain (low-level kernel optimization) that has nothing to do with abstract puzzle-solving. When the same structural pattern — inspect, plan, implement, evaluate, with persistent memory and stall detection — produces gains in two unrelated domains, that’s a signal the architecture itself is doing real work, not overfitting to one benchmark’s quirks.

The loop, stripped down

If you want the shape of it without NVIDIA’s branding, here’s roughly what the control loop looks like:

state = load_persistent_memory(task_id)

while not solved and budget_remaining:
    inspection = inspect(environment, state)
    if stall_detector.flags(state.trajectory):
        plan = replan(inspection, discard=state.trajectory)
    else:
        plan = plan_next_step(inspection, state)

    result = implement(plan, tools=environment.scoped_tools)
    evaluation = evaluate(result, environment)

    state = update_memory(state, evaluation)
    if evaluation.stalled:
        stall_detector.record(state.trajectory)

The part most teams skip when they build their own agent loops is the stall detector and the explicit discard-and-replan branch. It’s tempting to let an agent keep iterating on its current approach because “it’s already made progress” — but that’s exactly the failure mode that burns budget without moving the needle. AVO’s supervision layer treats “this trajectory isn’t converging” as a first-class signal that triggers a structural reset, not just another retry.

Why this matters if you’re running your own agent fleet

I run a chunk of my own operational work — this blog’s publishing pipeline, a couple of daily digests — through scheduled agents, and the AVO result is a useful gut-check against a habit I see in a lot of teams (including, occasionally, my earlier self): reaching for a bigger or newer model when an agent underperforms, instead of asking whether the harness around it is the actual bottleneck. A 30-point score gap closed entirely through scaffolding is a strong argument that “upgrade the model” should not be the default first move when an agent task is failing. The cheaper diagnostic question is: does this agent have persistent memory across the task, does anything detect when it’s stuck, and is its toolset scoped to what the task actually needs — or is it working with the equivalent of a blank terminal and a prayer?

The uncomfortable implication for anyone selling “just use our frontier model” as the whole pitch is that AVO’s result suggests a meaningful chunk of what looks like model capability is actually harness capability, and that gap is currently wide open for anyone willing to build the scaffolding properly instead of waiting for the next model release to close it for them.

Export for reading

Comments