Golden set
A golden set is a versioned reference set of real cases — real documents, real quotes, real expected outputs — that a system's model-dependent behavior is tested against before changes ship. Change the prompt, the model version, or the retrieval strategy, run the set, and see what improved and what quietly broke. In practice it combines a protected holdout, rotated hard cases, and samples drawn from production.
Why it matters
Model behavior shifts in ways code review cannot catch. For a dealer's quoting system, a golden set built from actual price books and quotes is what separates "the extraction seems fine" from "the extraction is measured" — and it is what makes an AI-assisted system maintainable by whoever comes after the original builders.
Common confusion
A golden set is not a one-time acceptance test — it runs on every change, and it grows a new case when production surprises someone, which is why versioning it matters. It also does not replace production monitoring: a reference set catches regressions before release; monitoring catches the drift and failures that only live traffic reveals.
Sources
Where we use it
The data-engineering context is Data engineering is the AI engineer most vendors don't hire; verification commitments are in the AI Policy.