On September 8, Inception Labs released Mercury 2.5, a diffusion-based language model hitting 1,107 tokens per second on standard NVIDIA GPUs — roughly 10x what a frontier autoregressive model manages on the same hardware. It scores 79% on GPQA Diamond and 77% on IFBench, putting it in the same intelligence tier as cost-optimized frontier models like Claude Haiku 4.5 and Gemini 3.5 Flash-Lite, while generating text through a fundamentally different mechanism than every chat model you’ve shipped against so far.

This is worth a technical lead’s attention not because “faster model” is news — every release claims that — but because diffusion generation changes where latency comes from in an agent pipeline, and that has architectural consequences.

Autoregressive vs. diffusion, in practice

Every GPT/Claude/Gemini-family model you’ve built against generates left-to-right: predict token N+1 given tokens 1..N, append it, repeat. Latency scales with output length because each token depends on the one before it — there’s a hard sequential dependency chain.

Mercury’s diffusion approach instead starts with a fully masked (noised) sequence and iteratively denoises the entire output in parallel across a fixed number of refinement steps, borrowing the masking-corruption process from image diffusion models but adapted for discrete tokens. Output length still matters for quality, but not for the number of sequential dependency hops — the whole sequence gets refined together.

The practical result: instead of ~100 tok/s wall-clock on a typical GPU, you get 1,000+. Not because the GPU got faster, but because the generation algorithm needs far fewer sequential passes to produce the same length of output.

import time
from inception import Client  # illustrative SDK shape

client = Client(api_key="...")

start = time.perf_counter()
response = client.chat.completions.create(
    model="mercury-2.5",
    messages=[{"role": "user", "content": "Refactor this function to be pure and add type hints."}],
    max_tokens=800,
)
elapsed = time.perf_counter() - start

print(f"{len(response.choices[0].message.content.split())} words in {elapsed:.2f}s")
# On Mercury 2.5: ~800 tokens in well under a second.
# The same request against a 100 tok/s autoregressive model: 8+ seconds.

Quality trails frontier autoregressive models by 5-15% on complex multi-step reasoning benchmarks, but is roughly at parity on structured output, translation, and template-shaped generation — the categories that dominate a lot of real agent tool-call traffic.

Where this actually matters: the latency budget of an agent loop

Here’s the architectural point. In a multi-step agent loop — plan, call tool, observe result, decide next step, repeat — the model’s generation latency compounds across every hop. A five-step agent loop at 2 seconds of generation latency per step is 10 seconds of pure model time before you even count tool execution and network round-trips. That’s the difference between an agent that feels responsive inside a CLI or IDE and one that feels like you’re waiting on a slow CI pipeline.

Diffusion models like Mercury 2.5 target exactly this pain point. If your bottleneck is “the model takes too long to decide what to do next,” not “the model isn’t smart enough to decide correctly,” a 10x generation speedup is a bigger lever than switching to a smarter-but-slower frontier model.

Where I’d route Mercury-class models in a real pipeline:

  • Structured intermediate steps: JSON tool-call formatting, code formatting/linting passes, diff generation, template filling — high-frequency, latency-sensitive, tolerant of a small quality gap.
  • Draft-then-verify patterns: generate a fast draft with Mercury, verify or repair with a slower frontier model only when the draft fails a check. This is the same speculative-decoding intuition applied at the pipeline level instead of the token level.
  • Interactive/IDE-embedded agents: autocomplete-adjacent tasks where users perceive anything over ~300ms as lag, regardless of how smart the underlying reasoning is.

Where I would not route it yet: the actual planning/reasoning step in an agentic loop where a wrong decision costs more than the latency you saved — multi-file refactors with subtle cross-file invariants, security-sensitive code review, anything where the 5-15% reasoning gap is the whole ballgame.

The bigger architectural signal

Diffusion LLMs have existed as a research curiosity for a couple of years (Google’s Gemini Diffusion, Mercury’s earlier releases); what’s different now is that Mercury 2.5 is closing the intelligence gap to genuinely-usable territory while keeping the throughput advantage intact. That combination — “good enough” reasoning at 10x the speed, at $0.04/$0.15 per million tokens — is the first time I’d tell a team to actually put a diffusion model in a production routing table next to their autoregressive default, not just in an experiment.

The lesson for anyone doing model routing in an agent stack: latency and intelligence are increasingly separable axes you can route on independently, the same way you already route on cost. If you’re still treating “which model” as a single decision per agent instead of a per-step routing decision, diffusion models are the forcing function to fix that.

Sources: AlphaSignal — Inception’s Mercury 2.5 Hits 1,107 Tokens per Second, Digital Applied — Mercury 2: Diffusion LLM at 1000+ Tokens/Second

Export for reading

Comments