← All writing

2 min readBuilding AI systems

A golden set in a week: evaluating LLM features without a research team

You don't need a research team to know whether a prompt change helped. A five-day plan to build a golden set from real inputs and run it on every change.

  • Evaluation
  • LLM systems
  • Quality

Most teams shipping an LLM feature can tell you it "seems better" after a prompt change. Few can tell you by how much, or what got worse. You don't need a research team to fix that. You need a golden set: a small, honest collection of real inputs, each with a clear idea of what a good output looks like.

Days 1–2: collect real inputs

Pull 50 to 100 real examples from logs, support tickets, or the people who will use the feature. Include the common, boring cases and the awkward ones: ambiguous requests, missing information, out-of-scope questions, and anything that has embarrassed you before.

Day 3: write down what good looks like

For each example, record the expected outcome in a form you can check: the right answer, the right source, the right category, or a short list of things a good response must and must not do. If the team can't agree on what good means, that's a product decision to settle now, not a model problem.

Day 4: pick checks that fit the task

  • Exact or rule-based checks wherever possible: the right label, a valid JSON shape, a required field present, a forbidden claim absent.
  • Reference checks for retrieval: did the expected source appear in the context?
  • A model-graded rubric for open-ended answers, written as specific yes/no questions and spot-checked against human judgment.
  • Cost and latency per example, recorded alongside quality.

Day 5: run it on every change

Run the set before and after each prompt, model, or retrieval change, and compare results per example, not just the average. A change that lifts the average by two points while breaking five previously correct answers deserves a closer look.

An evaluation set doesn't have to be big. It has to be honest, and it has to run every time.

Common traps

  • Testing only the happy path.
  • Letting the set go stale while the product changes.
  • Trusting a model-graded score you've never checked against a person.
  • Optimising the average and missing regressions on the cases that matter most.