Evaluating an agentic RAG system for regulatory audit support
The golden-goose eval battery fails a run when any metric gate, the non-leakage check or a coverage floor is missed. No current recorded run of the battery is published. The repository does not publish figures, and this page reports none. It describes what the battery checks and the gate values set in the code.
- Published
- 2026-10-28 (draft)
- Revised
- 2026-11-05: recorded results removed
- Topic
- Enterprise: regulated quality
- Source
- darblinko/golden-goose-qa, main branch
- Results
- None published
Scope
golden-goose is a LangGraph agent that answers questions about a medical device quality management system. It uses hybrid retrieval in Qdrant, a Neo4j knowledge graph and a Postgres record warehouse. Text answers cite document ID and revision, with a page for PDF sources.
The intended-use statement scopes the system to decision support during regulatory audit preparation and live audit response. The user opens the cited revision and confirms each claim before relying on it. A human writes the response to the auditor. The system is read-only over quality records and is not a 21 CFR Part 11 electronic record or signature system.
All evaluation data is synthetic. The repository generates a QMS for the fictional Halvern Medical infusion pump family, together with a golden question set and seeded defects.
What the battery runs
gg-eval battery builds the same dependency stack as the API, forces the external tools offline, and runs four evals over one golden question file: retrieval, end-to-end answers, routing and reranker noise. The run passes when its list of failures is empty. Gate failures and coverage failures go into the same list.
CI runs the battery on the smoke subset of the golden set for each push and pull request. A scheduled nightly workflow rebuilds the corpus, ingests it and runs the battery on the full golden set.
Gates set in the code
The values below are the defaults of BatteryThresholds. A run fails if any one is missed.
| Gate | Value in code | Note recorded in the code |
|---|---|---|
| Retrieval recall | at least 0.80 | |
| Grounded answer rate | at least 0.80 | |
| Sentence support | at least 0.50 | Lexical token coverage; ADR 0019 records the choice of a lexical metric. |
| Citation precision, citation recall | at least 0.50 each | Provisional floors, set below expected performance and to be tightened once the nightly run establishes a baseline. Both average over grounded answers only. |
| Count answer correctness | 1.00 | Applied only when the question set has checked count questions. Each has one right answer, a register count, so any wrong checked answer fails the run. |
| Routing accuracy | at least 0.90 | |
| Noise degradation | at most 0.15 | |
| Non-leakage | no leaks | A scoped question that surfaces an excluded document or revision fails the run regardless of every score. |
The coverage floors guard the denominators. The recall, grounded, routing and noise gates average over the questions that survive selection, so a question set that lost questions would leave every gate green. The floors fail the battery when coverage itself regresses, independently of the scores. The full set and the smoke set have separate floors.
Results
No current recorded run is published. The repository README states that evaluation results depend on the synthetic example Action Priority table and on the provider configuration, and that the repository does not publish figures. Its Quickstart lists the commands that generate the corpus, ingest it and run the battery in a local environment.
Limits of the method
- Fake providers. Without API keys every provider is a deterministic fake. The regex layer is then the only router, the generator is extractive and the verifier does not judge entailment. Under these conditions the battery measures retrieval, routing and the citation path. It does not measure a hosted LLM's answer quality.
- Dense retrieval under fakes. Keyless embeddings are SHA-256 pseudo-vectors, so recall comes from the BM25 branch and the reranker stand-in.
- Lexical support. Sentence support measures token overlap with the evidence. A strict xfail test records that it cannot detect an actor swap.
- Synthetic corpus. The generator that writes the documents also writes the golden questions. The README states that results on this corpus do not transfer to a real QMS without a new golden set.
- Scope. The system is read-only decision support. Its answers point to controlled documents and do not replace them, and the eval does not test the user's verification step.