I’ve been building distributed systems for over fifteen years. Message queues, microservices, event-driven pipelines — I’ve seen most failure modes at least once. But nothing quite prepared me for the particular chaos of multi-agent AI systems in production.

Last year my team deployed a multi-agent workflow for an enterprise client: an orchestrator agent, three specialist sub-agents, a tool-calling layer, and a memory service stitched together with async message passing. On paper, it was elegant. In production, it humbled us within the first week.

Here’s what I learned, broken down into the failures we hit and the patterns that actually fixed them.

Why Multi-Agent Systems Fail

Retry Storms

The first failure was the most embarrassing because it was so avoidable. One sub-agent hit a transient 503 from an upstream API. It retried. Fine. But our orchestrator was also watching for timeouts and issued its own retry. Meanwhile, the original sub-agent’s second attempt succeeded — but the orchestrator’s retry spawned a duplicate sub-agent. Now we had two agents doing the same work, writing conflicting outputs to shared state.

Within minutes, a single 503 had cascaded into a retry storm that exhausted our token budget and corrupted three task results.

This happens because in multi-agent systems, retry logic exists at multiple layers simultaneously — the LLM SDK, your agent runner, your orchestrator, and sometimes the tool layer itself. Without coordination, each layer retries independently and multiplies the load.

Context Loss Between Agents

The second failure was subtler. Our orchestrator passed a task to a specialist agent with a minimal handoff: just the task ID and a one-line description. The specialist agent had no access to the reasoning chain that led to the task, no prior context, and made a decision that was locally reasonable but globally wrong.

We’d optimized for small message size and created an agent that was flying blind. Context is not overhead — it’s safety rail.

Tool Timeouts with No Circuit Breaker

Our agents called external tools: a search API, a code execution sandbox, a vector database. When the code sandbox started responding slowly under load, agents would hang waiting for the timeout (30 seconds by default). Meanwhile, the orchestrator interpreted the silence as the agent still working, and queued more tasks. By the time timeouts started firing, we had a backlog of 40+ hung agents, each holding a context window and a thread.

There was no circuit breaker. There was no graceful degradation. There was just a pile of spinning agents eating compute.

Patterns That Work

Circuit Breakers for Agent Tool Calls

The first thing we added was a circuit breaker around every external tool call. The pattern is not new — it’s straight from Michael Nygard’s Release It! — but it’s rarely applied at the agent layer.

import time
import asyncio
from enum import Enum
from dataclasses import dataclass, field
from typing import Callable, Any

class CircuitState(Enum):
    CLOSED = "closed"      # Normal operation
    OPEN = "open"          # Failing, reject calls
    HALF_OPEN = "half_open"  # Testing recovery

@dataclass
class CircuitBreaker:
    name: str
    failure_threshold: int = 5
    recovery_timeout: float = 30.0
    half_open_attempts: int = 2

    _state: CircuitState = field(default=CircuitState.CLOSED, init=False)
    _failure_count: int = field(default=0, init=False)
    _last_failure_time: float = field(default=0.0, init=False)
    _half_open_successes: int = field(default=0, init=False)

    async def call(self, fn: Callable, *args, **kwargs) -> Any:
        if self._state == CircuitState.OPEN:
            if time.monotonic() - self._last_failure_time > self.recovery_timeout:
                self._state = CircuitState.HALF_OPEN
                self._half_open_successes = 0
            else:
                raise RuntimeError(
                    f"Circuit '{self.name}' is OPEN — tool call rejected"
                )

        try:
            result = await fn(*args, **kwargs)
            self._on_success()
            return result
        except Exception as exc:
            self._on_failure()
            raise exc

    def _on_success(self):
        if self._state == CircuitState.HALF_OPEN:
            self._half_open_successes += 1
            if self._half_open_successes >= self.half_open_attempts:
                self._state = CircuitState.CLOSED
                self._failure_count = 0
        else:
            self._failure_count = 0

    def _on_failure(self):
        self._failure_count += 1
        self._last_failure_time = time.monotonic()
        if self._failure_count >= self.failure_threshold:
            self._state = CircuitState.OPEN


async def call_tool_with_backoff(
    circuit: CircuitBreaker,
    tool_fn: Callable,
    *args,
    max_retries: int = 3,
    base_delay: float = 1.0,
    **kwargs,
) -> Any:
    for attempt in range(max_retries):
        try:
            return await circuit.call(tool_fn, *args, **kwargs)
        except RuntimeError as e:
            # Circuit is open — no point retrying
            if "OPEN" in str(e):
                raise
            # Transient failure — exponential backoff with jitter
            if attempt < max_retries - 1:
                delay = base_delay * (2 ** attempt) + (0.1 * attempt)
                await asyncio.sleep(delay)
            else:
                raise

Every external tool call now goes through a circuit breaker. When a tool starts failing, the circuit opens after five failures, rejects calls fast for 30 seconds, then lets through probe requests to check recovery. This alone eliminated the hung-agent backlog problem.

Idempotent Task Design

The retry storm problem required a different fix: idempotent tasks with stable task IDs.

Every task the orchestrator issues now carries a deterministic ID derived from its inputs. If an agent picks up a task and the orchestrator also retries, both invocations carry the same task ID. The task store deduplicates on write — the second write is a no-op.

The rule is: an agent task must produce the same observable outcome whether it runs once or ten times. This means writing outputs to keyed storage (not appending), using upsert semantics everywhere, and making tool calls idempotent where possible (or at least safe to replay).

Graceful Degradation Over Hard Failure

When a specialist agent is unavailable or times out, the orchestrator now has an explicit degradation path. Instead of failing the entire workflow, it flags the sub-task as degraded, records what context it had, and continues with a reduced capability mode. A document analysis workflow might skip the deep semantic search and fall back to keyword matching. An answer is returned, marked as lower confidence.

This is a product decision as much as a technical one. Work with your stakeholders to define acceptable degradation paths before you need them in production at 2am.

Naive vs. Resilient Architecture

Dimension	Naive Agent System	Resilient Agent System
Retry logic	Per-layer, uncoordinated	Centralized, circuit-broken
Task identity	Ephemeral, anonymous	Stable idempotent IDs
Context handoff	Minimal (task ID only)	Structured context bundle
Tool failure	Hang until timeout	Fast-fail via circuit breaker
Degradation	Binary success/fail	Tiered degradation paths
Observability	LLM logs only	Distributed trace per agent turn

The naive system works fine at low load and low failure rates — which is exactly why it passes staging. Production load surfaces every assumption you made when things were going well.

Production Checklist for Multi-Agent Systems

Before you ship an agentic workflow to production, I want you to be able to answer yes to every one of these:

• Idempotency: Can every agent task be replayed safely with the same task ID?
• Circuit breakers: Is every external tool call protected by a circuit breaker with explicit open/half-open/closed states?
• Retry coordination: Is retry logic defined at exactly one layer, or are multiple layers coordinating explicitly?
• Context contracts: Does every agent receive a structured context bundle, not just a task ID?
• Timeout budgets: Does every agent call have an explicit timeout, and does the orchestrator account for it in its own timeout budget?
• Degradation paths: Is there a defined, tested fallback for every agent that can fail?
• Distributed tracing: Can you trace a single user request end-to-end across all agent invocations?
• Token budget monitoring: Do you have alerts when an agent workflow approaches its context limit?
• Dead letter handling: What happens to a task that fails all retries? Is it logged, queued for review, or silently dropped?
• Load testing with failure injection: Have you run your agent system with simulated tool failures and measured the blast radius?

Lessons Learned

Multi-agent systems fail in ways that are qualitatively different from standard microservices. The failures are nondeterministic, context-dependent, and often only visible after several steps of latent damage. The LLM at the center does not throw exceptions — it just makes worse decisions when given bad context.

The good news is that the underlying reliability patterns are not new. Circuit breakers, idempotency, exponential backoff, graceful degradation — these have been battle-tested in distributed systems for years. What’s new is knowing where to apply them in an agentic architecture and accepting that the LLM layer is not magic: it fails like any other remote call, and it needs to be treated that way.

Build for failure from day one. Your agents will thank you at 3am when they handle the outage gracefully instead of calling you about it.