Mitchell Foutz All writing

Evaluating an agentic RAG system for regulatory audit support

The golden-goose eval battery fails a run when any metric gate, the non-leakage check or a coverage floor is missed. No current recorded run of the battery is published. The repository does not publish figures, and this page reports none. It describes what the battery checks and the gate values set in the code.

Published
2026-10-28 (draft)
Revised
2026-11-05: recorded results removed
Topic
Enterprise: regulated quality
Source
darblinko/golden-goose-qa, main branch
Results
None published

Scope

golden-goose is a LangGraph agent that answers questions about a medical device quality management system. It uses hybrid retrieval in Qdrant, a Neo4j knowledge graph and a Postgres record warehouse. Text answers cite document ID and revision, with a page for PDF sources.

The intended-use statement scopes the system to decision support during regulatory audit preparation and live audit response. The user opens the cited revision and confirms each claim before relying on it. A human writes the response to the auditor. The system is read-only over quality records and is not a 21 CFR Part 11 electronic record or signature system.

All evaluation data is synthetic. The repository generates a QMS for the fictional Halvern Medical infusion pump family, together with a golden question set and seeded defects.

What the battery runs

gg-eval battery builds the same dependency stack as the API, forces the external tools offline, and runs four evals over one golden question file: retrieval, end-to-end answers, routing and reranker noise. The run passes when its list of failures is empty. Gate failures and coverage failures go into the same list.

CI runs the battery on the smoke subset of the golden set for each push and pull request. A scheduled nightly workflow rebuilds the corpus, ingests it and runs the battery on the full golden set.

Gates set in the code

The values below are the defaults of BatteryThresholds. A run fails if any one is missed.

GateValue in codeNote recorded in the code
Retrieval recallat least 0.80
Grounded answer rateat least 0.80
Sentence supportat least 0.50Lexical token coverage; ADR 0019 records the choice of a lexical metric.
Citation precision, citation recallat least 0.50 eachProvisional floors, set below expected performance and to be tightened once the nightly run establishes a baseline. Both average over grounded answers only.
Count answer correctness1.00Applied only when the question set has checked count questions. Each has one right answer, a register count, so any wrong checked answer fails the run.
Routing accuracyat least 0.90
Noise degradationat most 0.15
Non-leakageno leaksA scoped question that surfaces an excluded document or revision fails the run regardless of every score.

The coverage floors guard the denominators. The recall, grounded, routing and noise gates average over the questions that survive selection, so a question set that lost questions would leave every gate green. The floors fail the battery when coverage itself regresses, independently of the scores. The full set and the smoke set have separate floors.

Results

No current recorded run is published. The repository README states that evaluation results depend on the synthetic example Action Priority table and on the provider configuration, and that the repository does not publish figures. Its Quickstart lists the commands that generate the corpus, ingest it and run the battery in a local environment.

Limits of the method