Golden set

A golden set is a fixed, curated collection of real cases — real documents, real quotes, real expected outputs — that a system's model-dependent behavior is tested against before any change ships. Change the prompt, the model version, or the retrieval strategy, run the golden set, and see what improved and what quietly broke.

Why it matters

Model behavior shifts in ways code review cannot catch. For a dealer's quoting system, a golden set built from actual price books and actual quotes is what separates "the extraction seems fine" from "the extraction is measured" — and it is the artifact that makes an AI-assisted system maintainable by whoever comes after the original builders.

Common confusion

A golden set is not a one-time acceptance test. Its value is that it runs on every change, forever — and that it grows a new case every time production surprises someone.

Where we use it

The data-engineering context is Data engineering is the AI engineer most vendors don't hire; verification commitments are in the AI Policy.