Most of the coverage of NVIDIA’s “Open Data for Agents” release framed it as a model-training story: here’s a big pile of trajectories, here’s how you fine-tune an agent on it. That’s true, but it undersells what’s actually useful about it if you’re not training foundation models — you’re running a team that builds internal tools on top of Claude Code, Cursor, or an in-house agent harness, and you have no systematic way to answer “is our agent actually getting better or worse over time.”
The datasets — Open-SWE-Traces (200K+ trajectories), SWE-Zero (318K trajectories), SWE-Hero (34K trajectories) — are worth reading not for the weights they’ll produce, but for the schema they standardize. That schema is a better internal eval format than most teams are currently using.
What’s actually in a trajectory
A trajectory in this format isn’t a prompt/response pair. It’s the full record of an agent session: the task description, every tool call the agent made in sequence, the tool’s return value, the agent’s reasoning between calls, and the terminal outcome (a patch, a test result, a failure). Open-SWE-Traces trajectories were synthesized running SWE-agent and OpenHands against real GitHub issues, with Minimax-M2.5 generating the “with thinking” traces and Qwen3.5-122B-A10B generating the “without thinking” comparison set — which itself is a useful design choice: it gives you paired data on how much a visible reasoning step changes tool-call accuracy on the same task.
That structure — task, ordered tool calls, intermediate state, terminal outcome — is exactly what you need to answer questions your team almost certainly can’t answer right now: which tool calls does our agent retry unnecessarily? Where does it hallucinate a file path instead of checking first? Which failure mode recurs across unrelated tasks?
Repurposing the schema for your own eval harness
You don’t need NVIDIA’s specific datasets to get value here — you need their trajectory schema applied to your own agent’s logs. Most agent harnesses (Claude Code included) already emit something close to this if you capture the full transcript rather than just the final output. Here’s a minimal version of that schema you can retrofit onto existing logs:
from dataclasses import dataclass, field
from typing import Literal
@dataclass
class ToolCall:
name: str
args: dict
result: str
succeeded: bool
@dataclass
class Trajectory:
task_id: str
task_description: str
tool_calls: list[ToolCall] = field(default_factory=list)
outcome: Literal["success", "partial", "failure"] = "failure"
outcome_detail: str = "" # e.g. test output, patch diff, error trace
def failure_signature(traj: Trajectory) -> str:
"""Collapse a failed trajectory to a comparable signature for clustering."""
failed_calls = [tc.name for tc in traj.tool_calls if not tc.succeeded]
return f"{traj.outcome}:{'>'.join(failed_calls[-3:])}" # last 3 failing tool calls
def cluster_failures(trajectories: list[Trajectory]) -> dict[str, int]:
from collections import Counter
sigs = [failure_signature(t) for t in trajectories if t.outcome == "failure"]
return dict(Counter(sigs).most_common(10))Run cluster_failures over a week of your agent’s session logs and you get a ranked list of recurring failure shapes — not “the agent failed 40 times,” but “37 of those 40 failures end in the same three-tool-call pattern: grep returns nothing, read guesses a wrong path, edit fails on a stale line number.” That’s an actionable bug report against your own tooling or prompt design, generated from data you already have but weren’t structuring.
Why the “with thinking” vs “without thinking” split matters more than it looks
The paired synthesis approach — same task, same tools, one run with visible reasoning traces and one without — is the part of this release I’d actually borrow for internal evaluation, independent of the datasets themselves. If you’re deciding whether to enable extended thinking, a planning step, or a “think before tool call” system prompt addition in your own agent, the honest way to measure the delta isn’t vibes — it’s running the identical task set both ways and diffing the trajectory outcomes, not just pass/fail but tool-call efficiency (fewer wasted calls, less path-guessing, cleaner failure recovery).
Most teams skip this because building the harness to run identical tasks twice and diff the traces feels like overhead. NVIDIA effectively published the reference implementation for that harness at dataset scale — you can build a much smaller version scoped to your own repo’s task distribution in an afternoon.
The limits worth being honest about
This is SWE-bench-style data — GitHub issue resolution on public repos, which optimizes for patch correctness against a known test suite. It doesn’t cover the failure modes that matter most in a lot of internal tooling contexts: multi-turn conversations with ambiguous human intent, tasks that touch proprietary internal APIs the model has never seen, or long-running sessions that span days rather than one sitting. Fine-tuning directly on Open-SWE-Traces will move the needle on generic code-fix tasks and tell you very little about your actual internal agent’s failure modes on your actual codebase.
That’s precisely why the schema, not the weights, is the reusable part. Public trajectory data trains a slightly better generic model. Your own trajectory data, captured in the same structured format, is the only thing that tells you where your specific agent deployment is actually breaking — and until now, most teams didn’t have a standard shape to capture it in. This release is as much a nudge toward “start logging your agent sessions as structured trajectories, not just final outputs” as it is a training corpus.
Sources: