Replay testing

Record a run once. Test against it forever — no API key, no network, no token spend.

Testing agents is normally miserable: the model is nondeterministic, calls cost money, and CI has no API key. Replay fixes all three.

Record one real run. Replay it against your current code, offline.

Record

Set JONAKI_RECORD to a directory and run your agent as usual:

JONAKI_RECORD=runs npx tsx --env-file=.env agent.ts

That writes runs/runs/<runId>.json containing both the trace and every model request/response pair. Commit it.

You can also force recording in code:

import { run } from "@rohhitrz/jonaki";

await run(agent, "book me a flight", { store, record: true });

The important part: tools still run

Only the model is replayed. Your tools execute for real.

That is what makes a recording a test rather than a snapshot. Change a tool's implementation, replay, and see exactly what breaks:

agent.ts

import { Agent, tool } from "@rohhitrz/jonaki";
import * as z from "zod";

export const toolCalls: { name: string; args: unknown }[] = [];

export const searchFlights = tool({
  name: "search_flights",
  description: "Search available flights.",
  input: z.object({ from: z.string(), to: z.string(), date: z.string() }),
  run: (args) => {
    toolCalls.push({ name: "search_flights", args });
    return JSON.stringify([
      { flight: "AI-802", priceUSD: 143 },
      { flight: "6E-511", priceUSD: 118 },
    ]);
  },
});

const agent = new Agent({
  name: "Travel",
  instructions: "Book the cheapest flight unless told otherwise.",
  tools: [searchFlights],
});

export default agent;

agent.test.ts

import { beforeEach, describe, expect, it } from "vitest";
import { replay } from "@rohhitrz/jonaki";
import agent, { toolCalls } from "./agent.js";

const FIXTURE = new URL("./runs/runs/booking.json", import.meta.url).pathname;

beforeEach(() => {
  toolCalls.length = 0;
});

describe("the recorded booking run", () => {
  it("runs with no API key at all", async () => {
    expect(process.env["ANTHROPIC_API_KEY"]).toBeUndefined();

    const result = await replay(FIXTURE, { agent });
    expect(result.status).toBe("completed");
  });

  it("really executes the tools", async () => {
    await replay(FIXTURE, { agent });
    expect(toolCalls.map((call) => call.name)).toEqual(["search_flights"]);
  });

  it("searches the route the user asked for", async () => {
    await replay(FIXTURE, { agent });
    expect(toolCalls[0]?.args).toMatchObject({ to: "Guwahati" });
  });
});

Why the loop needed no changes

ReplayProvider implements the same Provider interface as the Anthropic, OpenAI, and Google adapters. To the agent loop it is just another backend.

Zero lines of the run loop know replay exists. That is the payoff of putting the model behind an interface: recording is a Provider that wraps another one, and replay is a Provider that reads from a file.

Divergence

If your code has moved on far enough that the recording no longer fits — the recorded model calls a tool you deleted — you get a clear failure rather than a confusing one:

JONAKI_REPLAY_DIVERGED
  runId:    run_wNKbuGgKMgH9
  step:     model call #2 (recorded turn 1)
  expected: search_flights, book_flight
  actual:   search_flights
  missing:  book_flight
  hint:     the recording calls `book_flight`, which the agent no longer
            exposes; restore the tool or re-record

Divergence is detected when:

the recorded response calls a tool the agent no longer offers, or
the run needs more model calls than were recorded.

Catch it explicitly if you want to assert on it:

import { JonakiError, replay } from "@rohhitrz/jonaki";

await expect(replay(FIXTURE, { agent: shrunkAgent })).rejects.toThrowError(
  expect.objectContaining({ code: "REPLAY_DIVERGED" }),
);

A regression that is not divergence

If a tool still exists but now throws, the replay completes and the failure shows up in the trace — which is usually what you want to assert on:

import { replay, tool } from "@rohhitrz/jonaki";

const result = await replay(FIXTURE, { agent: agentWithBrokenTool });

const failed = result.steps.find(
  (step) => step.type === "tool" && step.status === "error",
);
expect(failed).toBeDefined();

Running a whole directory

npx jonaki test runs/runs --agent ./agent.ts

replaying 1 recording(s) against Travel

  ✔ booking.json                 3 model · 2 tools · 683ms

1/1 passed

It exits 1 on failure, so it drops straight into CI:

.github/workflows/test.yml

- run: npx jonaki test runs/runs --agent ./agent.ts

The --agent module must export the agent as its default export.

Comparing against the recording

replay() also hands back the run it was compared against:

import { replay } from "@rohhitrz/jonaki";

const result = await replay(FIXTURE, { agent });

result.recorded.steps.length;  // what happened when it was recorded
result.steps.length;           // what happened just now

What is stored

runs/runs/booking.json

{
  "createdAt": 1785612806000,
  "updatedAt": 1785612809700,
  "state": { "runId": "run_wNKbuGgKMgH9", "status": "completed", "steps": [] },
  "recording": {
    "version": 1,
    "runId": "run_wNKbuGgKMgH9",
    "agentName": "Travel",
    "turns": [
      {
        "index": 0,
        "provider": "anthropic",
        "model": "claude-sonnet-4-5",
        "request": {
          "messages": [],
          "system": "Book the cheapest flight.",
          "toolNames": ["search_flights"]
        },
        "response": {
          "text": "",
          "toolCalls": [],
          "usage": { "inputTokens": 786, "outputTokens": 96, "costUSD": 0.0038 },
          "finishReason": "tool_calls",
          "raw": null
        }
      }
    ]
  }
}

state holds the full RunState — the trace. recording.turns holds the model exchanges.

AbortSignal is stripped, the bulky provider raw payload is dropped, and API keys are scrubbed — so a recording is safe to commit.

On this page