Three frontier models shipped within ten days of each other this July: GPT-5.6 Luna on July 1, Claude Sonnet 5 on July 5, Grok 4.5 on July 8. Each wins on a different axis — Sonnet 5 leads coding benchmarks at 82.1% SWE-Bench Pro, GPT-5.6 Terra scores highest on multi-step agent reliability at 97.2%, Grok 4.5 is fastest at ~140ms p95 TTFT and cheapest at $2/$6 per million tokens. The model battle is over. The routing battle is just starting.

I’ve been through this before with cloud instances. You don’t pick one EC2 size and scale everything to it. You route workloads to the right compute tier. The same logic now applies to LLMs, and teams that haven’t built a routing layer are leaving 40-80% of their AI budget on the table.

Why Single-Model Default Fails at Scale

When a team first adopts an LLM API, they pick one model — usually whatever scored best on the benchmark they read — and send everything to it. This works fine in pilots. It breaks at production scale.

The math is straightforward. If you’re processing one million API calls monthly on Claude Sonnet 5 at $3/million input tokens (after the introductory period ends August 31, 2026), with an average of 2,000 tokens per request, you’re spending ~$6,000/month on input alone. But here’s the real problem: most of those requests don’t need Sonnet 5. They’re simple classification calls, summarization of short text, or status checks that a $0.25/million model handles just as well.

In practice, 70-80% of production LLM calls fall into what I call “commodity” territory — tasks where a Haiku-tier model performs at 95%+ of the flagship quality. You’re paying flagship prices for commodity work.

The Three-Tier Architecture

The pattern that works in production is a three-tier cascade, with a routing layer in front:

Request → Router → [Nano | Standard | Frontier]
                        ↓          ↓          ↓
                  Haiku 4.5   Sonnet 5   GPT-5.6 Terra
                   $0.25/$1   $3/$15     $5/$20

Tier 1 — Nano (Haiku 4.5 or equivalent): Simple classification, short summarization, intent detection, slot filling, anything under ~500 tokens with structured output. Target: 70-80% of your requests.

Tier 2 — Standard (Claude Sonnet 5, Sonnet 4.6): Complex reasoning, code generation under 200 lines, document synthesis, nuanced tone matching. Target: 15-25% of requests.

Tier 3 — Frontier (GPT-5.6 Terra, Claude Opus 4.8): Multi-step agentic tasks requiring >95% reliability, complex code reviews, tasks where failure is expensive. Target: 5-10% of requests.

A 75/20/5 split across these tiers cuts a $6,000 Sonnet-only bill to roughly $1,800 — a 70% reduction with no user-visible quality loss on the 75%.

Building the Router

The router is the hardest part. You have three options, in increasing sophistication:

Option 1: Rule-Based Routing (Start Here)

The simplest approach is a rules engine based on request metadata you already have:

def route_request(request: LLMRequest) -> ModelTier:
    # Hard rules based on task type
    if request.task_type in ["classify", "intent", "slot_fill"]:
        return ModelTier.NANO
    
    if request.task_type in ["agent_step", "complex_code_review"]:
        return ModelTier.FRONTIER
    
    # Token budget estimation
    estimated_tokens = estimate_tokens(request.messages)
    if estimated_tokens < 800 and request.structured_output:
        return ModelTier.NANO
    
    if estimated_tokens > 8000 or request.requires_tool_calls > 5:
        return ModelTier.FRONTIER
    
    return ModelTier.STANDARD

This gets you 60-70% of the cost savings immediately, with zero machine learning required. Ship this first.

Option 2: Classifier-Based Routing

Train a small classifier (a fine-tuned Haiku or a lightweight BERT model) that predicts the minimum model tier needed for a given request. The RouteLLM paper (ICLR 2025) showed this approach achieves 85%+ cost reduction while preserving 95% of quality on standard benchmarks.

class RouterClassifier:
    def __init__(self, model_path: str):
        self.classifier = load_model(model_path)
    
    def predict_tier(self, request: str) -> tuple[ModelTier, float]:
        logits = self.classifier(request)
        tier = ModelTier(logits.argmax())
        confidence = logits.softmax()[tier]
        
        # Fall up if confidence is low
        if confidence < 0.8:
            tier = ModelTier(min(tier.value + 1, ModelTier.FRONTIER.value))
        
        return tier, confidence

The key insight: when the classifier is uncertain, always route up, not down. A wrong route to a lower tier costs quality; a wrong route to a higher tier costs money but preserves correctness.

Option 3: Cascading with Quality Gates

For high-stakes workloads, run the request on the cheaper model first and evaluate the output quality before returning it. If the output fails a quality check, escalate to the next tier.

async def cascade_request(request: LLMRequest) -> LLMResponse:
    for tier in [ModelTier.NANO, ModelTier.STANDARD, ModelTier.FRONTIER]:
        model = get_model(tier)
        response = await model.complete(request)
        
        quality_score = evaluate_quality(response, request.quality_criteria)
        
        if quality_score >= request.min_quality_threshold:
            record_routing_decision(tier, quality_score, cost=model.cost)
            return response
        
        if tier == ModelTier.FRONTIER:
            # Final tier, return best available
            return response
    
    return response

This adds latency (two model calls on the ~30% that escalate), but for workloads where quality matters more than speed, the cascade guarantees you never ship a bad response from a cheap model.

Workload-Model Matching in 2026

Based on current benchmarks, here’s the mapping I use for my team:

WorkloadBest ModelReason
Code generation (<200 lines)Claude Sonnet 582.1% SWE-Bench Pro
Multi-step agentsGPT-5.6 Terra97.2% tool-chain reliability
Latency-sensitive chatGrok 4.5140ms p95 TTFT
Classification / triageHaiku 4.595%+ quality at 12x lower cost
Document synthesisClaude Sonnet 5Strong at structured extraction
Complex reasoningClaude Opus 4.8Maximum depth, use sparingly

The “use sparingly” on Opus 4.8 is real advice. I have teams spending 40% of their AI budget on Opus for tasks that Sonnet handles identically. Reserve the frontier tier for tasks where failure is genuinely costly.

Prompt Caching: The Other 40%

Routing isn’t the only lever. If your application sends the same system prompt, document context, or conversation history with each request — and most do — prompt caching is the second-highest-impact optimization available.

Cache reads on Anthropic’s API cost approximately 10% of the standard input rate. For a system prompt of 2,000 tokens sent with every request, that’s $0.27/million tokens saved on every cached call. At scale, this compounds quickly.

def build_cached_request(user_message: str, system_prompt: str) -> dict:
    return {
        "model": "claude-sonnet-5",
        "system": [
            {
                "type": "text",
                "text": system_prompt,
                "cache_control": {"type": "ephemeral"}  # Cache this block
            }
        ],
        "messages": [{"role": "user", "content": user_message}]
    }

In practice, routing + caching together consistently deliver 60-75% cost reduction on production workloads. The combination matters: caching reduces per-token cost at each tier, and routing reduces which tier you hit.

Observability: What to Measure

A routing layer without observability is flying blind. The metrics that matter:

Routing distribution — What % of requests hit each tier? If 50% hit Frontier, your router is misconfigured. Target 70-80% on Nano.

Quality escalation rate — For cascade routing, what % of requests escalate from one tier to the next? >40% escalation on Nano→Standard suggests your Nano tier isn’t well-specified.

Cost per request by tier and task type — Which task types are surprisingly expensive? These are candidates for prompt engineering or tier re-classification.

Quality by tier — Track user satisfaction, downstream task success, or automated eval scores by tier. A 2% quality drop that saves 60% on cost is usually worth it; a 20% drop isn’t.

@dataclass
class RoutingMetric:
    request_id: str
    tier_used: ModelTier
    tier_intended: ModelTier
    escalated: bool
    cost_usd: float
    quality_score: float
    task_type: str
    latency_ms: int

# Log every routing decision
metrics_collector.record(routing_metric)

The Rollout Strategy

Don’t flip the switch all at once. Here’s the rollout sequence I recommend:

  1. Week 1-2: Shadow mode — run the router, log routing decisions, but continue sending all requests to your current model. Validate that routing decisions make sense.

  2. Week 3-4: Route only low-risk task types (classification, triage) to lower tiers. Monitor quality metrics closely.

  3. Week 5-6: Expand to higher-volume standard tasks. Track escalation rate and quality scores.

  4. Week 7+: Full routing with cascade fallback for high-stakes workloads. Review cost dashboard weekly.

Each stage should show measurable cost reduction without quality regression. If quality drops, tighten the router’s confidence thresholds before proceeding.

The Model Landscape Is Different Now

The meaningful shift in July 2026 isn’t that any single model got dramatically better. It’s that the frontier became plural. You have three competitive models with meaningfully different strengths within a week of each other. Teams that pick one and ignore the others are making the same mistake as picking a single cloud region and ignoring the rest.

Multi-model routing is not a nice-to-have optimization anymore. At the scale most product teams are operating, it’s the difference between AI being profitable and AI being a cost center that leadership questions every quarter.

Build the router. Start with rules. Ship it. Then make it smarter.

Export for reading

Comments