ClickHouse’s Agentic SQL Benchmark Shows the Bottleneck Moved Upstream of SQL

ClickHouse just published results from a benchmark most teams building analytics agents should read closely, because it overturns an assumption a lot of eval design is still built on: that the hard part of text-to-SQL is getting the SQL right.

The benchmark, open-sourced as data-agent-mnist, took 201 real analytics questions drawn from the production traffic of DWAINE, ClickHouse’s own internal analytics agent, and ran them against a synthetic-but-faithful reconstruction of their data warehouse. 28 models, proprietary and open-weight, were each given up to 60 turns per question to explore the schema, write queries, inspect results, and arrive at an answer. There is no gold SQL query to match against. Correctness is scored against the result set a provider-diverse ground-truth committee and an LLM-as-jury panel agreed on, with each judge excluded from assessing models in its own family.

That design matters. Most text-to-SQL benchmarks reward matching a reference query. This one rewards getting the right number, however the agent gets there. It is closer to how an analytics copilot actually gets used, and that is exactly why the failure data is worth taking seriously.

SQL execution errors are close to solved

ClickHouse breaks down incorrect answers into five failure modes: FM1 (no attempt), FM2 (wrong plan), FM3 (wrong data), FM4 (wrong implementation), and FM5 (runtime error). FM5, the category that corresponds to SQL that simply fails to execute, is now close to absent across the board. Only four of the 28 models recorded any runtime errors at all, and only one, Gemma 4 31B, exceeded a 2% rate. For every frontier model in the lineup, writing SQL that actually runs against the schema is a solved problem.

That is a real result, not a footnote. For years, a meaningful share of text-to-SQL failures were syntactic: wrong column reference, malformed join, a dialect quirk the model didn’t know. ClickHouse’s data says that era is largely over for current frontier models operating inside a multi-turn agent loop, where they can see error messages and self-correct.

The dominant failure happens before any SQL is written

Here is the number that should change how teams think about agentic analytics evals: FM2, wrong plan, is by far the most common failure mode, accounting for 53 to 82% of failures for every model with a substantial failure count. This is not a tie-breaker between two close failure modes. It is the dominant cause of task failure, and it happens upstream of SQL generation entirely.

A wrong plan means the agent picked the wrong tables, the wrong join path, or the wrong filters before it ever wrote a query. The SQL it produces from that plan can be syntactically perfect, execute cleanly, and still answer a question nobody asked. A model can score zero on FM5 and still fail the task, because the thing it got wrong was never expressible as a SQL error in the first place. It was a reading-comprehension and schema-navigation error.

ClickHouse’s own framing of this is direct: SQL syntax and execution are no longer bottlenecks for modern models; for agentic analytics, data modeling and schema context matter far more. That is a specific, falsifiable claim grounded in their own benchmark data, not a general observation about AI maturing.

The diagram above lays out where the benchmark’s 201 questions actually break down. The agent first has to plan: pick the right tables, the right joins, the right filters, based on reading the question against a schema it has to discover for itself. Only after that plan is set does it write SQL, with up to 60 turns to iterate. The outcome split is stark: SQL runtime errors are close to zero, while wrong-plan failures make up the majority of everything that goes wrong. The bottleneck sits entirely in that first box, not the second.

Why this matters for teams building analytics copilots

If you are building or evaluating an agentic BI tool internally, this result has a direct consequence for where eval budget should go. A lot of text-to-SQL eval harnesses still spend most of their effort checking whether generated SQL parses, executes, and matches an expected query shape. ClickHouse’s data suggests that effort is now mostly wasted on a problem current frontier models have already solved on their own, especially when given a multi-turn loop where they can see and react to execution errors.

What is worth measuring instead is plan quality, independent of whether the resulting SQL runs:

• Did the agent select the correct tables for the question, including ones that require a non-obvious join to reach?
• Did it apply the filters the question actually implied (the right date range, the right status flag, the right entity scope), rather than filters that are plausible but wrong?
• Did it correctly resolve ambiguous business terms against the schema, rather than guessing which column a term like “active user” or “net revenue” maps to?
• When the plan was wrong, did the agent have any way to notice, short of the final number looking obviously off?

None of these are SQL-correctness questions. They are questions about whether the agent understood what to compute before it tried to compute it. An eval suite that only checks the final query or the final result set, without separating out plan-level mistakes from execution-level mistakes, will keep reporting the wrong thing: it will look like a SQL-generation problem when it is actually a comprehension problem, and teams will keep tuning the wrong stage of the pipeline.

Practically, that argues for annotating failures the way ClickHouse did, by stage, not just by pass or fail. A failure logged only as “wrong answer” tells you nothing about whether to invest in better schema documentation and semantic layers, or in SQL-generation prompting and tool design. A failure logged as “wrong plan, used the orders table instead of the fulfilled_orders view” tells you exactly where to spend the next engineering hour.

The editorial takeaway

The headline model in ClickHouse’s leaderboard, Claude Fable 5.1, led correctness at 76.6% across the 201 questions, with a cheaper model like DeepSeek V4 Flash reaching 65.7% at roughly 52 times lower cost to run the full benchmark. Those are useful numbers for picking a model. But the more durable finding is the failure-mode breakdown, because it says something about where the field’s remaining problem actually is, independent of which model currently tops the leaderboard.

Teams that keep optimizing SQL-generation accuracy, cleaning up prompt templates for dialect quirks, adding retry loops for syntax errors, are polishing a surface that frontier models have already made smooth. The actual remaining work is making the agent’s understanding of the question, and its map of the schema, reliable enough that the plan is right before a single line of SQL gets written. That is a harder problem to eval, because it requires judging intent rather than output, but it is the one the data says is actually unsolved.

Source: ClickHouse, “The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents”, data-agent-mnist benchmark, 201 questions, up to 60 turns per question, 28 models evaluated.