Prompt change records and the limits of their keyword checks
In llm-trading-lab a prompt change is recorded as the text before, the text after, their diff, and a rationale that names the trigger, the expected behavior change and the observation that would show the change was wrong. A validator checks each record against seven convention rules. The three rationale checks are keyword searches, and the unfilled template passes all three, so a passing record shows the form of a rationale. It does not show that a falsifier was written or that any test was run.
- Published
- 2026-10-28 (draft)
- Topic
- Quant trading: LLM operations
- Revision
7e4ec74
Scope: one package, no prompt text
llm-trading-lab is research tooling for LLM trading agents. Its project page covers the whole repository: the call gate, the walk-forward runner and the leakage audit. This article covers one package, src/prompt_history/, which writes a change record for each prompt edit, validates a tree of records against the convention, and counts how many records pass each check.
Neither the repository nor this article contains live prompt text. The one example record in the repository is fictional, and this article does not reproduce its prompt snapshots.
The record format: one folder, four files, one changelog entry
python -m prompt_history record takes the prompt name, a short slug and the two versions of the prompt file. It writes one folder named YYYY-MM-DD_<prompt_name>_<slug> holding four files, and the rationale starts as a template with five headed sections. It refuses a prompt name that is not snake_case, a slug outside 2 to 5 kebab-case words, an edit where before and after are identical, and a folder that already exists.
| Field | Stored in | What it is for |
|---|---|---|
| Date, prompt name, slug | Folder name | Dates the change and names the prompt it changed. The index counts records per month and per prompt from the folder name alone. |
| Before | before.md | The full prompt as the agent saw it before the change. |
| After | after.md | The full prompt after the change. With the before snapshot it lets the validator recompute the diff. |
| Diff | diff.patch | A unified diff of the two snapshots: the changed lines without the rest of the prompt. The validator compares its added and removed lines with a diff it recomputes, so a snapshot edited after the record was written is reported. |
| Trigger | rationale.md | Template text: "Which review, incident or measurement prompted this change." |
| Enforced rule | rationale.md | Template text: "The rule the added or changed text is meant to impose." No check reads this section. |
| Expected change | rationale.md | Template text: "What the agent should now do differently, stated so it can be counted in the logs." |
| Falsifier | rationale.md | Template text: "The observation over the next sessions that would show the change did not work." |
| Changelog link | rationale.md cross-references and the changelog | Ties the record to the changelog entry that announced the change. The validator passes a record when its folder name appears anywhere in the changelog text. |
The expected change and the falsifier answer different questions. The expected change is a count that shows the edited prompt altered what the agent does. The falsifier is an observation that shows the altered behavior did not serve its purpose. Inference: a change can produce its expected change and still be falsified, which is the case the falsifier exists to catch.
The changelog entry is a separate document with its own template: Files, Trigger, Change, Why, Contradiction check, Validation and Status are required, and Outcome is filled in after at least one session. A field counts when its bold label is present; the parser does not check that text follows the label. The repository does not define the Contradiction check field; the example entry fills it with "none found".
Lifecycle of a record
flowchart TD obs["Observation
session review of the
daily LLM call trace"] --> trig["Trigger
review, incident
or measurement"] trig --> edit["Prompt edited"] edit --> rec["Record written
before.md, after.md,
diff.patch, rationale template"] rec --> fill["Rationale filled in
trigger, rule,
expected change, falsifier"] fill --> log["Changelog entry
Files, Trigger, Change, Why,
Contradiction check,
Validation, Status"] log --> run["Next sessions run
on the new prompt"] run --> check{"Falsifier
observed?"} check -->|"no"| keep["Change kept"] check -->|"yes"| revert["Change reverted
or revised"] keep --> outcome["Outcome field filled
in the changelog entry"] revert --> outcome revert -.->|"new record"| edit rec -.-> val["validate and index
convention checks and counts"]
Each LLM call made by the live loop is written to a daily markdown trace with its prompt name, payload, raw response, parsed result and token usage, and two review apps show the traces on the price chart. A session review that leads to a prompt change starts a record.
Code covers three steps: writing the record, validating a tree of records, and indexing it into counts. The rationale, the changelog entry and the falsification check are written and done by a person. No code in the repository reads a falsifier or evaluates it against the traces. The two branches after the falsifier check are drawn from the example record, whose falsifier ends with the instruction to revert; the repository does not define what the Outcome field must contain.
The seven convention checks
Each check is a rule in validate.py. The index runs the validator on every record folder and counts the records that show none of the problems belonging to a check.
| Check | A record passes when |
|---|---|
follows_naming_convention | the folder name matches YYYY-MM-DD_<prompt_name>_<slug> |
has_all_four_files | before.md, after.md, diff.patch and rationale.md all exist |
diff_matches_snapshots | the added and removed lines of diff.patch equal those of a diff recomputed from the snapshots; a missing snapshot or patch fails |
names_trigger | rationale.md contains "trigger" in any case |
states_expected_change | rationale.md contains "expected" |
states_falsifier | rationale.md contains "falsif" |
linked_from_changelog | the folder name appears in the changelog text |
The three rationale checks also fail when rationale.md is missing.
What a failed check means
Each item below is an inference from the field definitions and the validator rules.
- No "expected" keyword. The rationale lacks the word, so the template heading was removed or not used. If such a rationale also states no expected change in other words, it gives no count to look for in the traces, and the record alone cannot show whether the edited prompt changed what the agent did.
- No "falsif" keyword. The rationale lacks the word and the template's Falsifier heading. If it also names no failure observation in other words, keeping or reverting the change rests on a judgment that the record does not capture.
- No "trigger" keyword in the rationale. The changelog template also has a Trigger field, so the trigger for such a record may sit in the linked changelog entry. The validator does not join records to entries, so this is not checked.
- No Validation field in a changelog entry. The entry does not say how the change was to be validated. The example entry fills the field with paper trading and a reminder to fill Outcome after at least one session.
- Not linked from the changelog. The record cannot be found from the changelog by folder name.
- Diff disagrees with snapshots. A snapshot was edited after the record was written, or the patch was made from different text. The diff then no longer describes the change exactly. The repository's test for this check edits
after.mdafter recording and expects this failure.
The keyword rule bounds what a passing check can show. A test in the repository records a change, leaves the rationale as the unfilled template and expects the validator, run without a changelog, to report no problem, because the template's own headings contain "Trigger", "Expected" and "Falsifier". For a record written from the template, a passing rationale check therefore does not show that the section was filled in. In the other direction, a rationale that states an expected change without the word "expected" fails.
A synthetic example record
The repository ships one fictional record and its changelog entry. The prompt snapshots are summarized below by size; the rationale sections are quoted from the example file.
Record folder: examples/prompt_history/records/2026-01-05_entry_review_require-volume-confirmation/
| File or section | Contents in the example |
|---|---|
before.md | 4 lines of fictional prompt text, not reproduced here. |
after.md | 7 lines of fictional prompt text, not reproduced here. |
diff.patch | One hunk: 3 lines added, 0 removed. |
rationale.md: Trigger | End-of-day review of 2026-01-02: two approved entries were taken on bars with falling volume and both reached the stop. |
rationale.md: What the new text enforces | An entry needs participation: the last 5 bars must average at least the 20-bar volume. |
rationale.md: Expected behavior change | Fewer approvals on low-volume drifts. In the trace logs, the share of WAIT decisions whose reason names the volume ratio should be above zero from the next session on. |
rationale.md: Falsifier | Over the next 10 sessions, approvals blocked by the volume rule hit their target at least as often as approvals the rule allowed. The rule then removes winners as often as losers and should be reverted. |
rationale.md: Cross-references | examples/prompt_history/CHANGELOG.md, entry of 2026-01-05. |
The expected change is a share that can be read from the traces from the next session. The falsifier names a window of 10 sessions, a comparison between blocked and allowed approvals, and the action to take when it is observed.
The changelog entry carries all seven required fields and a Prompt history line with the record's path. It has no Outcome, so the validator reports the outcome as pending and still counts 0 problems:
$ python -m prompt_history validate \
--root examples/prompt_history/records \
--changelog examples/prompt_history/CHANGELOG.md
changelog '2026-01-05 08:45 ET: require volume confirmation in entry review': outcome pending 0 problem(s)
Every compliance check passes on the example tree, and the 9 tests of the package pass.
Limits of this result
- Keyword checks. The rationale checks look for three words. An unfilled template passes them, and a rationale that uses other words fails them.
- No outcomes. The index counts Outcome labels. It does not count falsifiers that were observed, changes that were reverted or changes that were kept. Whether any stated falsifier was ever checked against the traces is not recorded in the repository.
- Counts only. The index writes counts per check, per month and per prompt name, and no per-record results. Overlaps between checks cannot be computed from its output.
- Records and changelog are not joined. The link check confirms a folder name appears in the changelog. It does not confirm the linked entry describes the same change.
- No prompt text. The record structure and the validator are published; the prompts are not. The example record is fictional.