When Your Agent Crashes at Step 87

Imagine you’ve just deployed a complex AI agent workflow: analyzing financial reports, synthesizing data from 12 different sources, generating forecasts, then sending a report to stakeholders. The workflow has 100 steps. You run it overnight.

In the morning, you check the logs and see it crashed at step 87.

The first question isn’t “why did it crash?” — it’s “what do I do now?” Restart from scratch? You lose 86 completed steps, burn another 3-4 hours of compute, and pay for the API calls again. Try to resume from step 87? With most current implementations, that’s not possible.

This is the core problem that durable execution solves — and why I’ve been paying close attention to it as an engineer who has built production AI workflows for the past two years.

Why Production AI Agents Can’t Just “Retry”

In the world of simple APIs, retry logic is enough. Request failed? Try again. But agentic workflows are fundamentally different.

An AI agent isn’t just calling one API — it’s a chain of stateful decisions. By step 50, the agent has read 30 files, queried 5 databases, called 8 external APIs, and built up a context window full of intermediate reasoning. When you restart from scratch, all that context is gone. The agent doesn’t remember what it “learned” in the previous 86 steps.

Worse, many steps in agentic workflows are not idempotent. Send an email notification at step 40? If you restart, it sends again. Create a ticket in Jira? You’ll have a duplicate. Write data to a database? Race conditions and inconsistencies.

This is the fundamental difference between toy demos and production systems: production doesn’t accept “just run it again.”

Three Core Patterns: Checkpointing, Replay, and Signed History

Durable execution isn’t a single feature — it’s a combination of three complementary patterns.

Checkpointing is saving execution state after each meaningful step. Not after every token generation, but after each logical step — each tool call, each external API call, each important state transition. A checkpoint needs to include: the result of that step, the inputs used, and enough context to resume from exactly that point.

Replay is the ability to fast-forward through execution history when resuming. When the agent crashes at step 87 and you restart, the system doesn’t re-execute steps 1-86 — it replays the checkpointed results. The agent receives the exact output of each prior step as if they just ran. Key insight: replay ≠ re-execute.

Signed Execution History adds security and integrity. Each step in history is signed with a cryptographic signature, ensuring the history hasn’t been tampered with, the replay represents exactly what happened, and you have a complete audit trail for compliance and debugging.

Diagrid Catalyst 2.0 implements all three patterns through the Dapr workflow engine. The interesting part is how it intercepts the agent runner lifecycle: instead of letting LangGraph, OpenAI Agents SDK, or Microsoft Agent Framework manage execution directly, Catalyst wraps that lifecycle and registers each step as a Dapr workflow activity. Result: durable execution without changing agent logic.

Non-Durable vs. Durable: The Practical Difference

Here’s a concrete example — a simplified financial analysis agent:

# NON-DURABLE: Stateless execution — restart from scratch on failure
class FinancialAnalysisAgent:
    def run(self, company_ticker: str):
        market_data = self.fetch_market_data(company_ticker)   # 45 seconds
        financials = self.fetch_financials(company_ticker)     # 30 seconds
        competitors = self.analyze_competitors(company_ticker) # 2 minutes
        # ... 97 more steps ...
        return self.generate_report(market_data, financials, competitors)

    # If this crashes at step 87, EVERYTHING restarts from step 1
    # Cost: 3+ hours of compute, hundreds of API calls, thousands of tokens
# DURABLE: Each step is a workflow activity with checkpoint
from dapr.ext.workflow import WorkflowActivityContext

class DurableFinancialAnalysisAgent:

    @workflow_activity  # Catalyst intercepts this
    def fetch_market_data(self, ctx: WorkflowActivityContext, ticker: str):
        # Catalyst checkpoints the result automatically
        # On replay, this function is NOT re-executed — result comes from checkpoint
        return self._fetch_from_api(ticker)

    @workflow_activity
    def fetch_financials(self, ctx: WorkflowActivityContext, ticker: str):
        return self._fetch_financials_api(ticker)

    def run(self, company_ticker: str):
        # Dapr workflow orchestrates the activities
        # Crash at step 87 → steps 1-86 replay from signed history
        # Only step 87+ needs to re-execute
        market_data = yield self.fetch_market_data(company_ticker)
        financials = yield self.fetch_financials(company_ticker)
        competitors = yield self.analyze_competitors(company_ticker)
        return yield self.generate_report(market_data, financials, competitors)

The difference isn’t just code structure — it’s the entire failure recovery model. With the non-durable agent, a network hiccup at step 87 = 86 steps wasted. With the durable agent, the same failure = resume from step 87, taking a few seconds.

Diagrid Catalyst 2.0 makes this transparent to LangGraph, OpenAI Agents SDK, and other frameworks by intercepting at the agent runner level — not at application code level. You don’t need to rewrite agent logic; the framework automatically wraps tool calls and LLM calls into workflow activities.

Real Trade-offs: When Does the Overhead Pay Off?

I want to be honest here: durable execution isn’t a silver bullet, and it has real costs.

Checkpointing overhead: Each step needs to serialize state and write to durable storage. For fast, simple agents running in seconds, this overhead can be significant. If your agent runs 10 steps in 5 seconds, checkpointing cost may exceed the benefit.

Replay logic complexity: Signed execution history requires deterministic replay. If your steps have non-deterministic side effects (random seeds, timestamps, external state), replay may produce different results — and the system needs to handle that. LLM calls are especially complex because temperature and sampling create non-determinism.

Storage and retention: Every workflow instance needs persistent storage for history. At high agent volume, this is meaningful infrastructure cost.

Debugging complexity: When agent behavior derives from replayed history rather than fresh execution, debugging traces become more complex. You need good tooling to distinguish “this step ran fresh” vs “this step was replayed.”

With those trade-offs in mind, when is durable execution actually worth it?

Decision Checklist for Tech Leads

After building multiple production agent systems, here’s the checklist I use:

You NEED durable execution if:

  • Workflow runs longer than 5 minutes: Anything that runs long enough has elevated risk of interruption from infrastructure issues, timeouts, or deploys.
  • Non-idempotent side effects exist: Sending emails, creating tickets, charging payments, writing database records — things you don’t want duplicated on retry.
  • Re-execution cost is high: If restarting from scratch costs more than $1 in API costs or more than 30 seconds of compute, durable execution overhead usually pays for itself.
  • Compliance requires audit trail: Signed execution history isn’t just resilience — it’s evidence for auditors.
  • Fan-out with many parallel steps: When an agent runs 10 parallel research tasks and one fails, you want to retry just that one, not all 10.
  • User-facing workflows with SLA: If users are waiting for a result and you have an SLA, “sorry, it crashed, run it again” is not an acceptable answer.

You can skip durable execution if:

  • Agent runs under 30 seconds: Fast agents rarely get interrupted and the overhead doesn’t justify itself.
  • Pure read-only operations: If the agent only reads data with no side effects, simple retry is sufficient.
  • Idempotent workflows: If every step is safe to re-execute, built-in framework retry logic is enough.
  • Low-cost, high-frequency operations: Chatbots, simple Q&A, real-time classification — latency matters more than resilience here.
  • Development and experimentation: In the prototype phase, simplicity usually matters more than durability.

Looking Forward

Diagrid Catalyst 2.0 is the first implementation I’ve seen that genuinely solves durable execution for AI agents transparently — no agent logic rewrite required, multiple frameworks supported, built on the battle-tested Dapr workflow engine.

But the more important point is the pattern itself. As AI agents are increasingly used for critical business workflows — not demos, but production systems with SLAs and compliance requirements — durable execution will become an expectation, not a nice-to-have.

If you’re building long-running AI workflows today and haven’t thought about failure recovery, revisit the checklist above. The question isn’t “will my agent ever crash?” — it’s “when it crashes, what do I want to happen?”

For production systems, the answer is always: resume from where it stopped, not restart from the beginning.

Thuận Lương is a Technical Lead with 15+ years of experience in .NET, cloud architecture, and AI systems. He writes about lessons learned building real production systems.

Export for reading

Comments