Here’s the failure mode nobody warned me about. I tweak an agent’s system prompt to fix one thing, it looks fine in a quick test, I ship it — and three other behaviors I’d carefully tuned quietly regress. Nobody notices until a user hits one. Agents are non-deterministic and their behavior is smeared across a prompt, a model, a temperature, and a tool list. You cannot eyeball that.

The fix is old news in software and overdue in agents: a test suite. In this world I call it an eval harness — a golden set of inputs and expected properties, run against every agent (and every workflow) on every change.

An eval is an input plus checks, run against a target

The unit is small. An eval names a target (an agent or a workflow), an input to send it, and a list of checks the output must satisfy. Running it is mechanical:

flowchart LR
  E["Eval: input + checks"] --> R["Run the target<br/>(agent or workflow)"]
  R --> O["Output + cost + latency"]
  O --> C{"Apply checks"}
  C -->|all pass| P["✅ pass"]
  C -->|any fail| F["❌ fail + which check"]
  P --> L["Log the run"]
  F --> L

A run captures more than the text: the output, the token cost, the latency, and any error. That lets my checks assert on behavior and budget — “answers correctly” and “under $0.02” and “under 3s.”

Two kinds of checks: deterministic and judged

Most properties are cheap to check with code. A few need judgment, so I borrow another model.

I reach for deterministic checks first — free, fast, unambiguous. json_valid alone catches a huge class of “the model wrapped it in prose” regressions. I use the judge only for genuinely fuzzy properties: “is this a polite refusal?”, “does it cite a source?”

The judge: a strict, independent grader

LLM-as-judge is a second model, independent of the agent under test, handed a rubric and the answer and asked for a single verdict. Keep it strict and structured:

You are a strict evaluator. Given a RUBRIC and an ANSWER, decide whether the
answer satisfies the rubric. Reply with a single word PASS or FAIL, then a
colon and one short sentence.

Temperature 0, a short token budget, PASS/FAIL first so parsing is trivial. The independence matters — never let an agent grade its own homework with its own prompt.

Run-on-change: the regression net

An eval suite you have to remember to run is an eval suite you won’t run. The leverage is automatic regression: when I edit an agent’s behavior-affecting fields — its prompt, model, tools, temperature — its evals fire in the background and flag any that flipped to failing.

flowchart LR
  Edit["Edit agent<br/>(prompt / model / tools)"] --> Q{"has evals?"}
  Q -->|yes| Run["run its golden set"]
  Q -->|no| Skip["skip"]
  Run --> Diff["report pass/fail<br/>+ Telegram on regressions"]

Now the golden set is a safety net stretched under every change, not a chore. A red result the moment I save is worth ten green results I ran a week ago.

Grade workflows, not just agents

Single-agent evals are table stakes. The interesting failures live in multi-agent workflows — an orchestrator delegating to specialists, or a node graph fanning out and back in. Those need the same treatment: run the whole workflow on a golden input, then check the final synthesized output.

The trick is to make the target polymorphic. An eval’s target_kind is agent or workflow; the harness runs the right executor and applies the identical checks to whatever comes out. If your eval runner only knows how to run single agents, your most complex — and most fragile — systems are the ones going ungraded. (That was a real gap in mine until recently; workflows now grade the same way agents do.)

The golden set is a living spec

The quiet payoff: my evals become the executable definition of what each agent is for. “Refuses off-topic requests.” “Always returns valid JSON.” “Cites at least one source.” “Costs under two cents.” Every check is a promise, and the suite is the promise I can run. New behavior starts as a new eval; fixing a bug starts by writing the eval that would have caught it.

You don’t need a platform to start — you need three columns (input, check, target) and the discipline to run them on every change. Agents earn trust the same way code does: not by looking right, but by passing.