Every agent demo I’ve ever seen — mine included — has the same structural flaw: it runs once, it works, and everyone in the room updates their belief about the agent’s reliability based on a sample size of one. Microsoft’s new open-source benchmark, ThinkingBox, exists specifically to kill that habit, and the numbers it produces are uncomfortable enough that they’re worth sitting with.

Grading the database, not the transcript

Most agent evals — including plenty I’ve built — score an agent by reading its own transcript: did it say it called the right tool, did it claim the ticket is resolved, did the final message look right. ThinkingBox throws that out. It’s a sandbox for tool-agent-user interaction with isolated, MCP-compatible tool sessions, and instead of trusting the agent’s account of what happened, it runs executable assertions against the actual backend state — the database rows — after the task completes.

That distinction matters more than it sounds like it should. An agent can produce a perfectly plausible closing message (“I’ve updated the customer’s address and confirmed the change”) while having silently failed to commit the write, retried against the wrong record, or partially applied the update and stopped. Transcript-based grading misses all three failure modes because it’s grading the agent’s narration, not its effect. State-based grading catches them by construction.

The benchmark: 507 tasks, five domains, 20 trials each

ThinkingBox-bench runs 507 policy-conditioned workflows across five business domains — retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. These aren’t toy tasks; they’re the kind of multi-step, stateful, policy-constrained workflows that actual internal tools automate: process a return under a specific refund policy, provision an account within a compliance boundary, resolve a support ticket that touches three backend systems.

The methodology is the part I’d steal even if I never touch Microsoft’s specific tool: each of the 507 tasks is run 20 separate times per model, and Microsoft reports both pass@1 (did it succeed on the first try) and pass^20 (did it succeed on every one of 20 tries). That second number is the one that matters for anything touching production state, because a support agent that’s right 65% of the time on a cold start is not a support agent you can leave unsupervised — you need it right closer to 100% of the time, consistently, across repeated runs of a nearly-identical task.

The results are the point

Across 12 proprietary and open-weight models, the best performer hit 65.36% pass@1 — a two-out-of-three first-try success rate on realistic business tasks. That same top model dropped to 25.25% pass^20. In other words: run the identical task twenty times, and the best available model gets it right on every single attempt only about a quarter of the time.

The failure mode Microsoft flags as most common isn’t crashes or thrown errors — it’s agents that “quietly did the wrong thing, or did part of the right thing and stopped.” That’s the failure mode that’s hardest to catch in a demo and easiest to miss in a code review of an agent transcript, because there’s nothing loud to notice. The agent doesn’t panic; it just quietly writes the wrong value and moves on.

Borrowing the pattern without the tool

You don’t need ThinkingBox itself to apply the idea. The pattern is:

def run_reliability_eval(agent, task, n_trials=20):
    successes = 0
    for i in range(n_trials):
        reset_backend_state(task.fixture)          # clean slate every trial
        agent.run(task.prompt)                       # let the agent act
        actual = query_backend_state(task.entity_id)  # check the system of record
        if assertions_pass(actual, task.expected_state):
            successes += 1
    return {
        "pass_at_1": successes >= 1,
        "pass_hat_n": successes == n_trials,   # every single trial succeeded
        "success_rate": successes / n_trials,
    }

The two changes that actually matter, independent of tooling: assert against the system of record instead of the agent’s self-report, and run the same task enough times to see the failure rate instead of a single anecdote. A CI eval suite that checks “did the agent call the expected tool” is necessary but nowhere near sufficient — it will happily pass an agent that calls the right tool with the wrong arguments and silently corrupts a record.

My take

I’ve written before about instrumenting agents for observability in production — this is the pre-production complement to that work. If you’re about to let an agent touch anything that resembles a system of record (billing, inventory, account state, ticket status), pass@1 on a handful of manual runs tells you almost nothing. Run it twenty times against the same scenario, assert on the actual data, and look at pass^20 before you trust it unsupervised. The gap between those two numbers, in Microsoft’s own results, is the gap between “worked in the demo” and “safe to ship.”

Paper & benchmark: arxiv.org/abs/2608.19741 · Code: github.com/microsoft/thinkingbox

Export for reading

Comments