Mitchell Foutz All writing

Prompt change records and the limits of their keyword checks

In llm-trading-lab a prompt change is recorded as the text before, the text after, their diff, and a rationale that names the trigger, the expected behavior change and the observation that would show the change was wrong. A validator checks each record against seven convention rules. The three rationale checks are keyword searches, and the unfilled template passes all three, so a passing record shows the form of a rationale. It does not show that a falsifier was written or that any test was run.

Published
2026-10-28 (draft)
Topic
Quant trading: LLM operations
Source
darblinko/llm-trading-lab
Revision
7e4ec74

Scope: one package, no prompt text

llm-trading-lab is research tooling for LLM trading agents. Its project page covers the whole repository: the call gate, the walk-forward runner and the leakage audit. This article covers one package, src/prompt_history/, which writes a change record for each prompt edit, validates a tree of records against the convention, and counts how many records pass each check.

Neither the repository nor this article contains live prompt text. The one example record in the repository is fictional, and this article does not reproduce its prompt snapshots.

The record format: one folder, four files, one changelog entry

python -m prompt_history record takes the prompt name, a short slug and the two versions of the prompt file. It writes one folder named YYYY-MM-DD_<prompt_name>_<slug> holding four files, and the rationale starts as a template with five headed sections. It refuses a prompt name that is not snake_case, a slug outside 2 to 5 kebab-case words, an edit where before and after are identical, and a folder that already exists.

FieldStored inWhat it is for
Date, prompt name, slugFolder nameDates the change and names the prompt it changed. The index counts records per month and per prompt from the folder name alone.
Beforebefore.mdThe full prompt as the agent saw it before the change.
Afterafter.mdThe full prompt after the change. With the before snapshot it lets the validator recompute the diff.
Diffdiff.patchA unified diff of the two snapshots: the changed lines without the rest of the prompt. The validator compares its added and removed lines with a diff it recomputes, so a snapshot edited after the record was written is reported.
Triggerrationale.mdTemplate text: "Which review, incident or measurement prompted this change."
Enforced rulerationale.mdTemplate text: "The rule the added or changed text is meant to impose." No check reads this section.
Expected changerationale.mdTemplate text: "What the agent should now do differently, stated so it can be counted in the logs."
Falsifierrationale.mdTemplate text: "The observation over the next sessions that would show the change did not work."
Changelog linkrationale.md cross-references and the changelogTies the record to the changelog entry that announced the change. The validator passes a record when its folder name appears anywhere in the changelog text.

The expected change and the falsifier answer different questions. The expected change is a count that shows the edited prompt altered what the agent does. The falsifier is an observation that shows the altered behavior did not serve its purpose. Inference: a change can produce its expected change and still be falsified, which is the case the falsifier exists to catch.

The changelog entry is a separate document with its own template: Files, Trigger, Change, Why, Contradiction check, Validation and Status are required, and Outcome is filled in after at least one session. A field counts when its bold label is present; the parser does not check that text follows the label. The repository does not define the Contradiction check field; the example entry fills it with "none found".

Lifecycle of a record

flowchart TD
  obs["Observation
session review of the
daily LLM call trace"] --> trig["Trigger
review, incident
or measurement"] trig --> edit["Prompt edited"] edit --> rec["Record written
before.md, after.md,
diff.patch, rationale template"] rec --> fill["Rationale filled in
trigger, rule,
expected change, falsifier"] fill --> log["Changelog entry
Files, Trigger, Change, Why,
Contradiction check,
Validation, Status"] log --> run["Next sessions run
on the new prompt"] run --> check{"Falsifier
observed?"} check -->|"no"| keep["Change kept"] check -->|"yes"| revert["Change reverted
or revised"] keep --> outcome["Outcome field filled
in the changelog entry"] revert --> outcome revert -.->|"new record"| edit rec -.-> val["validate and index
convention checks and counts"]

Each LLM call made by the live loop is written to a daily markdown trace with its prompt name, payload, raw response, parsed result and token usage, and two review apps show the traces on the price chart. A session review that leads to a prompt change starts a record.

Code covers three steps: writing the record, validating a tree of records, and indexing it into counts. The rationale, the changelog entry and the falsification check are written and done by a person. No code in the repository reads a falsifier or evaluates it against the traces. The two branches after the falsifier check are drawn from the example record, whose falsifier ends with the instruction to revert; the repository does not define what the Outcome field must contain.

The seven convention checks

Each check is a rule in validate.py. The index runs the validator on every record folder and counts the records that show none of the problems belonging to a check.

CheckA record passes when
follows_naming_conventionthe folder name matches YYYY-MM-DD_<prompt_name>_<slug>
has_all_four_filesbefore.md, after.md, diff.patch and rationale.md all exist
diff_matches_snapshotsthe added and removed lines of diff.patch equal those of a diff recomputed from the snapshots; a missing snapshot or patch fails
names_triggerrationale.md contains "trigger" in any case
states_expected_changerationale.md contains "expected"
states_falsifierrationale.md contains "falsif"
linked_from_changelogthe folder name appears in the changelog text

The three rationale checks also fail when rationale.md is missing.

What a failed check means

Each item below is an inference from the field definitions and the validator rules.

The keyword rule bounds what a passing check can show. A test in the repository records a change, leaves the rationale as the unfilled template and expects the validator, run without a changelog, to report no problem, because the template's own headings contain "Trigger", "Expected" and "Falsifier". For a record written from the template, a passing rationale check therefore does not show that the section was filled in. In the other direction, a rationale that states an expected change without the word "expected" fails.

A synthetic example record

The repository ships one fictional record and its changelog entry. The prompt snapshots are summarized below by size; the rationale sections are quoted from the example file.

Record folder: examples/prompt_history/records/2026-01-05_entry_review_require-volume-confirmation/

File or sectionContents in the example
before.md4 lines of fictional prompt text, not reproduced here.
after.md7 lines of fictional prompt text, not reproduced here.
diff.patchOne hunk: 3 lines added, 0 removed.
rationale.md: TriggerEnd-of-day review of 2026-01-02: two approved entries were taken on bars with falling volume and both reached the stop.
rationale.md: What the new text enforcesAn entry needs participation: the last 5 bars must average at least the 20-bar volume.
rationale.md: Expected behavior changeFewer approvals on low-volume drifts. In the trace logs, the share of WAIT decisions whose reason names the volume ratio should be above zero from the next session on.
rationale.md: FalsifierOver the next 10 sessions, approvals blocked by the volume rule hit their target at least as often as approvals the rule allowed. The rule then removes winners as often as losers and should be reverted.
rationale.md: Cross-referencesexamples/prompt_history/CHANGELOG.md, entry of 2026-01-05.

The expected change is a share that can be read from the traces from the next session. The falsifier names a window of 10 sessions, a comparison between blocked and allowed approvals, and the action to take when it is observed.

The changelog entry carries all seven required fields and a Prompt history line with the record's path. It has no Outcome, so the validator reports the outcome as pending and still counts 0 problems:

$ python -m prompt_history validate \
    --root examples/prompt_history/records \
    --changelog examples/prompt_history/CHANGELOG.md
changelog '2026-01-05 08:45 ET: require volume confirmation in entry review': outcome pending
0 problem(s)

Every compliance check passes on the example tree, and the 9 tests of the package pass.

Limits of this result