How to Build Deterministic Agent Tests with Recorded LLM Transcripts and Pytest

Quick answer

For a repeatable regression check, keep capture and replay as separate jobs. Capture a representative run, serialize its events, and use Pytest’s tmp_path to check that the written transcript can be read back with its events intact.

Review the captured transcript before saving it as a stable, version-controlled replay fixture. Preserve call identifiers, tool names, and arguments, plus results linked to calls when the run has multiple exchanges; the exact transcript format should suit your agent.

In the replay test, use monkeypatch to replace the model-call function the agent actually uses with a stub that returns recorded responses. Then compare the observed ordered tool events and arguments with the expected sequence, and assert the final result separately.

Pytest restores the patch after the test, and its assertion output can help you spot a mismatch.

Step-by-step guide

Starting point and requirements

Start with a Python agent whose model-call boundary is visible to tests: identify the function or client method it calls, and make sure your test can replace that reference. Pytest’s monkeypatch fixture can substitute a test implementation and restore the original afterward; patch the reference your application uses.

Choose a representative task that exercises the tool behavior you want to protect. For an OpenAI function-calling run, note that a response can contain zero, one, or multiple calls.

Preserve each call’s identifier, name, and JSON-encoded arguments, and record tool results linked to calls when the run continues across multiple model exchanges. Confirm your client exposes these fields before designing the transcript.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation.

Define the transcript events

Use a project-specific transcript format rather than assuming there is one standard cassette schema. Keep events in run order, and preserve each requested call’s identifier, tool name, and JSON-encoded arguments; an OpenAI function-call response can contain zero, one, or several calls.

For a multi-step run, record each tool result and link it to the matching call identifier. A compact representation could look like this:

[
  {"type": "tool_call", "id": "call_1", "name": "lookup", "arguments": "{\"key\":\"item-7\"}"},
  {"type": "tool_result", "call_id": "call_1", "content": "found"}
]

Adapt field names and result content to your client, but keep enough information to replay the exchange and later compare which tool ran, with what arguments, and in what order.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation.

Capture and check transcript serialization

In capture mode, run your representative agent task and serialize the resulting events under Pytest’s tmp_path. This gives the test an isolated temporary directory rather than leaving a transcript in your project.

import json

def test_capture_serializes_events(tmp_path):
    events = capture_run()  # Use your project's capture entry point.
    transcript = tmp_path / "transcript.json"
    transcript.write_text(json.dumps(events, indent=2), encoding="utf-8")

    saved = json.loads(transcript.read_text(encoding="utf-8"))
    assert saved == events

The read-back comparison checks that serialization preserved the captured events, including the call details and any linked tool results defined in your transcript format. Keep this capture-and-serialization check separate from replay tests, which should load reviewed, stable transcript fixtures.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation.

Keep reviewed transcripts for replay

After the capture-and-serialization test succeeds, review a representative transcript’s ordered calls, arguments, and linked results. Put the approved file in a stable location such as tests/fixtures/agent_transcripts/ and commit it with the test code.

Keep this fixture separate from capture output. Capture tests should write to tmp_path and check the read-back; replay tests should load the reviewed fixture instead of generating a fresh transcript.

That way, a changed transcript is a deliberate baseline update, not an automatic replacement during testing.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation.

Replay without a live model call

Load the reviewed transcript and select the recorded model responses in the order your agent expects them. Patch the model-call reference used by the agent—not just the client’s original import—and return responses in the same shape the agent normally receives.

def test_replay(monkeypatch):
    responses = iter(load_recorded_responses())

    def replay_call(*args, **kwargs):
        try:
            return next(responses)
        except StopIteration:
            raise AssertionError("Agent made an unexpected model call")

    monkeypatch.setattr(agent_module, "call_model", replay_call)
    result = agent_module.run_agent("sample task")

Replace the example module, function, loader, and task with your project’s equivalents. The test should reach the replay stub without contacting the model API; if agent tools make external calls, stub those separately.

Pytest restores the patched reference when the test ends.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation.

Assert tool behavior and final result

Build the observed tool-event list from your agent’s recorded events, keeping the calls in execution order. Compare tool names and parsed arguments so differences in JSON formatting alone do not count as behavioral drift.

import json

observed = [
    (event["name"], json.loads(event["arguments"]))
    for event in tool_events
]
expected = [("lookup", {"key": "item-7"})]
assert observed == expected
assert result == "found"

Adapt tool_events and result to your agent’s outputs. The final-result assertion is separate, so a failure shows whether the tool sequence or the user-facing outcome changed.

Pytest’s assertion report helps identify mismatched values; review any transcript change before updating the approved fixture.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation; How to write and report assertions in tests | pytest documentation.

Check the regression suite

Run the capture and replay checks separately. The capture test should write to tmp_path, read the transcript back using the same explicit encoding, and pass only when the saved events equal the captured events.

For replay, confirm the agent uses the patched model-call function rather than making a live model request. The test should pass when the observed tool calls match the expected sequence and normalized arguments, and the final result matches its expected value.

If an assertion fails, use Pytest’s comparison details to locate a changed event, order, name, argument, or result. Review the difference before changing a stable transcript fixture; a serialization pass alone does not establish that the transcript represents the intended behavior.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation; How to write and report assertions in tests | pytest documentation.

Diagnose replay failures and transcript drift

When a replay assertion fails, start with Pytest’s comparison diff. Check whether the first mismatch is a missing or extra event, a changed call order or tool name, different parsed arguments, or a changed final result; these point to different behavior changes.

If the stub was not used, patch the model-call reference the agent actually calls, which may be an imported reference in its own module. Pytest restores the patched reference after the test, so a live request suggests the patch target or test path needs checking.

Treat a newly captured transcript as evidence to review, not an automatic fixture update. Compare its calls, arguments, and linked tool results with the approved version, then update the stable fixture only when the change is intentional and reflects the behavior you want to preserve.

Sources: Function calling | OpenAI API; How to monkeypatch/mock modules and environments | pytest documentation; How to use temporary directories and files in tests | pytest documentation; How to write and report assertions in tests | pytest documentation.

Next steps

A useful replay suite has two distinct checks: one confirms that capture and serialization preserve the recorded events, while the other runs the agent against a reviewed transcript without making a live model call. Together, they let you check both the saved data and the behavior it is meant to reproduce.

If a replay fails, inspect the assertion difference for a changed event, order, tool name, argument, or final result, and confirm the stub replaced the model-call reference your agent uses. Update a stable fixture only after reviewing the transcript and deciding the change is intentional; then rerun the capture and replay checks.