2 min readBuilding AI systems
A golden set in a week: evaluating LLM features without a research team
You don't need a research team to know whether a prompt change helped. A five-day plan to build a golden set from real inputs and run it on every change.
Most teams shipping an LLM feature can tell you it "seems better" after a prompt change. Few can tell you by how much, or what got worse. You don't need a research team to fix that. You need a golden set: a small, honest collection of real inputs, each with a clear idea of what a good output looks like.
Days 1–2: collect real inputs
Pull 50 to 100 real examples from logs, support tickets, or the people who will use the feature. Include the common, boring cases and the awkward ones: ambiguous requests, missing information, out-of-scope questions, and anything that has embarrassed you before.
Day 3: write down what good looks like
For each example, record the expected outcome in a form you can check: the right answer, the right source, the right category, or a short list of things a good response must and must not do. If the team can't agree on what good means, that's a product decision to settle now, not a model problem.
Day 4: pick checks that fit the task
- Exact or rule-based checks wherever possible: the right label, a valid JSON shape, a required field present, a forbidden claim absent.
- Reference checks for retrieval: did the expected source appear in the context?
- A model-graded rubric for open-ended answers, written as specific yes/no questions and spot-checked against human judgment.
- Cost and latency per example, recorded alongside quality.
Day 5: run it on every change
Run the set before and after each prompt, model, or retrieval change, and compare results per example, not just the average. A change that lifts the average by two points while breaking five previously correct answers deserves a closer look.
An evaluation set doesn't have to be big. It has to be honest, and it has to run every time.
Common traps
- Testing only the happy path.
- Letting the set go stale while the product changes.
- Trusting a model-graded score you've never checked against a person.
- Optimising the average and missing regressions on the cases that matter most.