The Problem: AI Regressions Are Invisible

Six months into shipping an AI-powered onboarding assistant, my team got a Slack message from customer success: “Users are confused by the product recommendations again.” Not a spike in errors. No 500s in the logs. Just a slow bleed of users who stopped converting, and a support team that had been quietly fielding complaints for three weeks.

We dug in. Turns out, a prompt change from a sprint earlier — one that looked like a harmless tone adjustment — had shifted the model’s output format just enough to break how the frontend rendered structured recommendations. The model still returned valid JSON in 90% of cases. But in the other 10%, it wrapped the payload in a markdown code block. Our parser silently swallowed it. Users saw a blank card.

Three weeks. Nobody caught it because we had no eval pipeline. We were flying blind.

This is the invisible regression problem with AI. Unlike a broken API endpoint, a degraded model output doesn’t throw exceptions. It just quietly gets worse, and you find out from angry users — or you don’t find out at all.

What Is an Eval Pipeline?

An eval pipeline is not a test suite. It’s a systematic process that answers one question: is my AI system still behaving the way I intend it to?

The full cycle looks like this:

1. Define criteria — what does “good” mean for your specific use case? Accuracy? Tone? Format compliance? Safety? You need to make this explicit.
2. Build a dataset — a curated collection of inputs with expected outputs or judgment criteria.
3. Run eval — execute your model against the dataset and score outputs programmatically.
4. Analyze — aggregate scores, flag regressions, surface examples that fail.
5. Iterate — tune prompts, adjust temperature, update the dataset, repeat.

The output isn’t a pass/fail binary. It’s a score over time. You’re watching a trend, not flipping a switch.

The 3-Tier Eval Framework

After running evals across several production AI systems, I’ve settled on a three-tier approach that balances speed, depth, and cost.

Tier 1 — Deterministic checks

Fast, cheap, and run on every PR. These catch the obvious structural failures:

• Is the response valid JSON when JSON is expected?
• Does the response length fall within acceptable bounds?
• Are required fields present and non-null?
• Does the output contain any hallucinated field names not in your schema?

Tier 1 should complete in seconds. If it fails, you don’t need a human or another LLM to tell you something is wrong.

Tier 2 — Model-as-judge

This is where you use a capable LLM — typically GPT-4o or Claude Sonnet — to evaluate the outputs of your production model on dimensions that can’t be expressed as rules: relevance, tone, factual plausibility, helpfulness.

Here’s a minimal Python implementation of this pattern:

import anthropic
import json

def judge_response(user_input: str, model_output: str, criteria: str) -> dict:
    client = anthropic.Anthropic()

    judge_prompt = f"""You are an expert evaluator for AI assistant outputs.

User input: {user_input}

Model output: {model_output}

Evaluation criteria: {criteria}

Score the output on a scale of 1-5 for each criterion below.
Return a JSON object with keys: relevance, accuracy, tone, overall, reasoning.
Be strict. A 5 means genuinely excellent, not just acceptable."""

    message = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content": judge_prompt}]
    )

    raw = message.content[0].text
    # Parse JSON from the judge's response
    return json.loads(raw)


# Example usage in your eval loop
results = []
for example in eval_dataset:
    scores = judge_response(
        user_input=example["input"],
        model_output=run_model(example["input"]),
        criteria="Accuracy of product recommendations. Tone should be professional but approachable. Response must not hallucinate product names."
    )
    results.append({**example, "scores": scores})

A few hard lessons here: your judge prompt is as important as your production prompt. Vague criteria produce noisy scores. Be specific about what you’re evaluating and what each score level means.

Tier 3 — Human review

You cannot automate everything, and you shouldn’t try. Tier 3 exists for edge cases, safety-sensitive outputs, and the nuanced quality that judges miss. But human review is expensive, so sample efficiently.

Use stratified sampling: take the bottom 5% by Tier 2 score, a random 2% from the middle, and any outputs that triggered anomaly flags. That’s usually enough to catch systemic problems without overwhelming your reviewers.

Building Your Eval Dataset

Start smaller than you think you need. Fifty well-curated examples beat five hundred random ones. Your dataset should contain three types:

Golden examples — the happy path. Representative inputs with known-good expected outputs. These establish your baseline.

Adversarial cases — inputs designed to break your model: ambiguous phrasing, edge-case inputs, prompts that historically caused hallucinations. If you’ve seen a failure in production, it belongs here.

Regression cases — every bug you’ve fixed, turned into a test. This is your memory. When something breaks and you fix it, add the input to the dataset. Never let the same bug surprise you twice.

You need at least 30 examples to get signal. Under 30, noise dominates. Over 200, you’re probably over-indexing on coverage before your pipeline is mature. Grow the dataset in response to production incidents, not preemptively.

Integrating Evals into CI/CD

Tier 1 and Tier 2 evals belong in your CI pipeline. Here’s a minimal GitHub Actions setup:

name: AI Eval

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/ai/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install anthropic pytest

      - name: Run Tier 1 deterministic checks
        run: python -m pytest tests/eval/tier1/ -v

      - name: Run Tier 2 model-as-judge eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python scripts/run_eval.py \
            --dataset eval/dataset.jsonl \
            --threshold 3.8 \
            --output eval_results.json

      - name: Check eval threshold
        run: python scripts/check_threshold.py eval_results.json --fail-below 3.8

The --threshold 3.8 is a score gate. If average Tier 2 score drops below 3.8 out of 5, the PR is blocked. Set your threshold based on your baseline — don’t guess. Run your eval on the last 10 stable commits, take the 10th percentile score, and use that as your floor.

Practical Lessons from Production

The eval dead zone is real. There’s a range of model quality that is “bad enough to hurt users but not bad enough to fail your evals.” Your evals are only as good as your dataset and criteria. If you haven’t seen a failure mode in production, you probably haven’t tested for it.

Judge models drift too. The LLM you use as a judge will be updated. Its scores will shift. Pin your judge model version and re-calibrate your thresholds when you upgrade it. Treat a judge model upgrade like a dependency major version bump.

Prompt changes need eval coverage, not just code review. A prompt is code. It touches the eval dataset just as much as any TypeScript file does. Make it a rule: no prompt change ships without running evals.

Latency and cost compound fast. At 100 examples with a Tier 2 judge, each eval run costs roughly $0.10–0.30 and takes 2–4 minutes. At 1000 examples, you’re looking at a 20-minute CI job and $2–3 per run. Design your dataset size with CI economics in mind, and cache judge calls aggressively on unchanged inputs.

Getting Started Tomorrow

You don’t need to build all three tiers at once. Here’s where to start:

Step 1: Pick one AI feature that’s live in production and matters. Write 30 test cases — 20 golden examples from real usage logs, 10 edge cases you’re nervous about. Store them in a JSONL file.

Step 2: Write a Tier 1 checker. Just format validation and field presence. Wire it to run locally with pytest. Run it against your current prompt. Get that green.

Step 3: Add one Tier 2 judge call for your most important quality dimension. Score your 30 examples. Record the baseline score. Now you have a number to defend.

Don’t measure everything. Measure what breaks users. Add the judge, get a baseline, and wire it to CI. The rest is iteration.

Invisible regressions are a product problem disguised as a technical one. An eval pipeline makes them visible. That’s the only goal.