All Green. Then Production Explodes.

It was a Friday afternoon when the alert fired. A payment service was processing duplicate charges — customers were being billed twice for the same order. The fix had shipped that morning after a clean CI run: 47 passing tests, zero failures. The agent-generated code looked solid. The PR had been reviewed. Everyone had moved on.

The bug was in idempotency handling. The agent had written a processPayment function that checked a deduplication key before charging. It had also written the test that verified this logic. The mock it used returned a clean null on the first call and {exists: true} on repeat calls — exactly the behavior the agent assumed the payment gateway would exhibit. The real gateway, under high concurrency, returned a 202 Accepted on the first call and required a separate status poll before the deduplication key was committed. The agent never knew this. Its mock was internally consistent. It was just wrong.

This is the verification gap.

Why Agent-Written Tests Are a Different Problem

When a human engineer writes a mock, they bring external knowledge to it. They have read the API docs, been burned by edge cases before, maybe even filed a support ticket about that race condition in the gateway. The mock reflects real-world friction, even imperfectly.

When an AI agent writes a mock, it draws from training data: documentation, Stack Overflow threads, example repositories. It constructs a model of how it believes an external system behaves — and that model is internally consistent. The agent does not know what it does not know. It cannot mock the undocumented behavior that your payment gateway exhibits under load, or the OAuth token refresh race that only appears when two requests arrive within 300 milliseconds of each other.

The result is that agent-written tests validate the agent’s own assumptions, not production reality. A green test suite tells you the agent’s mental model is self-consistent. It tells you almost nothing about whether the code will survive contact with real systems.

This is fundamentally different from the usual testing problem of “tests that don’t cover enough cases.” Here, the tests cover the cases — they just cover the wrong cases, because the mock is wrong.

What This Looks Like in Code

Here is a simplified TypeScript example of the kind of mock an agent typically generates for a token refresh flow:

// Agent-generated mock — looks correct, misses a real edge case
class MockAuthClient {
  private tokenExpiry: number = Date.now() + 3600_000;
  private refreshCount: number = 0;

  async getToken(): Promise<string> {
    if (Date.now() < this.tokenExpiry) {
      return "valid-token-abc123";
    }
    return this.refresh();
  }

  private async refresh(): Promise<string> {
    this.refreshCount++;
    this.tokenExpiry = Date.now() + 3600_000;
    return `refreshed-token-${this.refreshCount}`;
  }
}

// Test passes cleanly
it("should refresh token when expired", async () => {
  const client = new MockAuthClient();
  jest.spyOn(Date, "now").mockReturnValueOnce(Date.now() + 4_000_000);
  const token = await client.getToken();
  expect(token).toMatch(/refreshed-token/);
});

The mock is sequential and synchronous in its mental model. In the real auth service, two concurrent requests arriving during a refresh window both trigger refresh calls, the second one invalidates the first token mid-flight, and one of those callers receives a 401 on a token it just received. The agent never modeled concurrency because the documentation did not mention it. The test passes. Production fails under load.

Three Things You Can Do About This

1. Shift verification into the inner loop — before the PR.

Do not wait for CI to be the first verification gate. Add a pre-commit or pre-push step that runs the agent’s output against a minimal real integration, not a mock. This can be as lightweight as a single contract assertion: call the real endpoint with a test credential, verify the response shape matches what the agent assumed. If the real endpoint is unavailable in the dev environment, you are already learning something important: the agent has been writing against a fiction.

Some teams are using lightweight “reality checks” — a small script the agent must run before submitting a PR that probes one or two critical external calls. The bar is not full integration coverage; it is “did the agent ever talk to the real thing?”

2. Use ephemeral environments with real dependencies for agent-generated code.

For agent-heavy workflows, spinning up a full environment with real (or staging) dependencies for every PR is no longer optional — it is the floor. Mocked environments are appropriate for unit-testing human-written logic. They are not appropriate as the primary validation layer for agent output, because the agent authored both the code and the environment model.

Tools like Testcontainers, Neon database branching, and Stripe’s test mode make this tractable without production risk. The cost of a 90-second ephemeral environment is far lower than a production incident.

3. Add contract tests between agent output and real APIs.

Pact-style contract testing — where you record the actual response shape from a real API call and assert that the code handles it correctly — works well as a standing safety net for agent-generated integrations. Unlike traditional contract tests, the point here is not to catch breaking API changes; it is to catch cases where the agent’s assumed API shape was never correct to begin with.

Schedule a nightly job that re-runs contract assertions against real endpoints. When it fails, you have found a place where the agent’s internal model diverged from reality before that divergence reached a customer.

The Takeaway

AI agents are genuinely useful at code generation. They are not reliable at generating the external knowledge that makes tests meaningful. Green CI on agent-written code is a weaker signal than green CI on human-written code — not because the agent writes worse logic, but because it writes its own reality into the test layer.

As a tech lead, your job is not to distrust the agent. It is to ensure the verification chain includes at least one step that the agent did not author. One real call. One contract assertion. One ephemeral environment. That is the gap you are closing.

If you have agent-generated code in production right now, the question worth asking this week is simple: when did that code last talk to the real system — not the mock the agent wrote for it?