How To Build An Evaluation Set For An Llm App Without Months Of Work?

If you’re shipping an LLM feature into production, you’re making a bet every time you change anything: prompt, model, retrieval config, tool policy, system message, guardrails, even your chunking strategy. The scary part is you usually don’t notice you broke something until a real user hits the exact edge case. And when that happens, the conversation logs become your incident report. Fun.

An evaluation set for an LLM app is how you stop shipping blind. It’s not a research project. It’s not a Kaggle competition. It’s basically a small, curated pile of real-ish user inputs plus a consistent way to judge the outputs so you can detect regressions and make tradeoffs intentionally.

I’ve watched teams “do evals” by eyeballing 20 random prompts in a notebook and declaring victory. It works until the first regression lands on your biggest customer. This post is the practical version: how to build a useful eval set fast, where people mess up, and what to do instead.

What an “Evaluation Set” Actually Is

An evaluation set is a set of test cases that represent real usage, plus a rubric (a scoring checklist) that lets you judge outputs consistently across runs. That’s it. It’s the bridge between “we changed stuff” and “we know what changed and whether it’s acceptable.”

What it is:

• A snapshot of the behaviors you care about: correct answers, refusal behavior, tone, format, groundedness, tool usage, latency, etc.

• A regression alarm system: “this new model/prompt/retriever broke our top flows.”

• A shared language for decisions: engineers, PMs, and stakeholders can point to the same results instead of vibes.

What it isn’t:

• A giant dataset with thousands of labels (unless you’re doing serious model training).

• A single “accuracy %” score that represents reality. (More on why that’s a trap later.)

• A perfect oracle of truth. In real products, “correct” often depends on policy, context, and user intent.

The goal is not academic purity. The goal is: catch failures early, measure progress, and ship with confidence.

The Golden Set: Definition, Purpose, and How Big It Should Be

What a “Golden Set” Means in LLM Apps

When people say “golden set,” they usually mean “the sacred spreadsheet nobody wants to touch.” Don’t do that.

In practice, a golden set is simply your highest-signal test cases: the ones that represent your core user journeys and your most expensive failures. It’s what you run every time you’re about to ship, and what you run automatically for LLM regression testing.

A good golden set has:

• Realistic inputs

  (from production logs when possible).

• The right context

  (retrieved docs, tool outputs, system constraints).

• Clear expected behavior

  (not necessarily a single exact answer).

• Tags

  so you can slice results (customer tier, intent type, language, risk level, etc.).

• rubric

  that turns “looks good” into “passes these checks.”

Here’s the mindset shift: in LLM apps, “expected output” is often not a string match. It’s behavioral: does it cite sources, does it refuse when needed, does it ask a clarifying question, does it call the right tool, does it avoid hallucinating.

That’s why the rubric matters more than people think.

How Big Should Your Golden Set Be?

If you try to label 1,000 examples on day one, you will create a multi-week project, everyone will hate you, and it will quietly die. I’ve seen it happen more than once.

Instead, build in tiers. This gives you fast feedback loops without pretending you can measure the universe.

Tier 1: Smoke

• “Did we obviously break the app?”

• on every prompt change, model switch, retriever tweak, tool policy update.

• your top 5 intents + the top 3 failure modes + 2–3 “must-not-break” customer workflows.

Tier 2: Release 

• “Is this safe enough to ship?”

• before releases, or nightly in CI depending on your setup.

• broader coverage + more edge cases, including multilingual, long context, tricky tool calls, and “user is confused” scenarios.

Tier 3: Deep 

• “How are we doing overall, and what should we fix next?”

• weekly or per sprint, often with more detailed analysis.

• long tail issues, rare but severe safety risks, and cases you’re actively improving.

If you’re a startup shipping fast: start with Smoke + a small Release set. Deep can come later. The point is to get to a working eval habit this month, not a perfect dataset next quarter.

A Simple Sampling Strategy That Works

You don’t need fancy stats to get value. You need coverage and realism.

Here’s a sampling strategy I’ve used repeatedly:

1. Pull 200–500 real user inputs

   from the last 2–4 weeks (after scrubbing PII).

2. Cluster by intent

   (manually is fine at this size“how-to,” “account issue,” “summarize doc,” “recommendation,” “policy question,” etc.

3. Within each intent bucket, choose

   • 2–3 most common queries (the boring bread-and-butter).

   • 2–3 high-friction queries (users re-ask, complain, or abandon).

   • 2–3 edge cases (ambiguous, adversarial, long, multi-turn).

4. Add a “Hall of Shame” bucket

   the top failures from incidents, escalations, or angry customer threads. These are disproportionately valuable.

This gets you a golden set that’s “small but sharp.” Most teams don’t need more complexity until they already have an eval pipeline running.

Build It Fast: A 7–14 Day Workflow That Doesn’t Melt Your Team

You can build a meaningful evaluation set for an LLM app in two weeks without sacrificing your whole roadmap. The trick is: don’t boil the ocean, and don’t over-label.

Day 1–2: Collect Real Inputs

• Export recent production queries (or internal dogfood logs if you’re pre-launch).

• Strip PII and secrets. If you can’t do this safely, synthesize structure, not content. Keep the same intent patterns.

• Capture the context that matters:

  • For RAG evaluation: store retrieved chunks + doc IDs + timestamps.

  • For tool use: store tool outputs (or mocks).

  • For multi-turn: store the conversation history leading up to the user input.

Common mistake: people save only user_input and then wonder why outputs vary wildly later. In real apps, the context is half the behavior.

Day 3–5: Bucket + Select Cases

• Create 6–12 intent buckets. Keep it simple.

• Identify “critical path” buckets (the ones tied to revenue, retention, or safety).

• Select the first version of your sets:

  • Smoke: 20–40

  • Release: +60–160

This is also where you assign severity:

• must not break (billing, legal, safety, top customer workflow).

• major UX failure (wrong, confusing, tool misuse).

• minor issues (tone, verbosity, formatting).

Severity matters because it stops you from treating “slightly wordy” the same as “made up a policy.”

Day 6–10: Label + Disagreement Loop

Labeling is where eval projects go to die unless you structure it.

• Start with a rubric (checklist) per intent type.

• Label in small batches (10–20 at a time).

• Do double-labeling for a subset (say 15–25%) to find rubric holes.

• When labelers disagree, don’t debate forever. Update the rubric with an example and move on.

A healthy disagreement loop produces:

• clearer rubrics,

• fewer “vibes-based” labels,

• faster labeling over time.

Day 11–14: Automate + Version

Now you turn the pile into something operational:

• Put test cases in version control (yes, git).

• Add metadata: tags, severity, intent, language, customer tier, “requires_tool,” “requires_rag,” etc.

• Wire a basic runner:

  • run your app on the set,

  • score outputs with your rubric (human or automated),

  • store results by version (prompt hash, model version, retriever config).

This is where LLM regression testing becomes real. Without versioning, you can’t answer: “when did this break?” or “did we actually improve anything?”

How to Label Answers Consistently (Rubrics That Don’t Drift)

Labeling is the part nobody wants to do. It’s also the part that determines whether your evals are signal or noise.

Use Checklists, Not Vibes

If your rubric is “good/bad,” you’ll get:

• inconsistent labels,

• endless debates,

• metrics that bounce around for no reason.

A rubric should be a checklist with observable criteria. Example:

Instead of:

• “Answer is helpful.”

Use:

• “Addresses the user’s question directly.”

• “If unknown, says so and asks a clarifying question.”

• “Does not invent facts not in context.”

• “Uses the required format (bullets/table/etc.).”

• “For RAG: cites sources or references retrieved docs.”

The best rubrics read a little boring. That’s a feature.

Write Rubrics Like Unit Tests (Rubric Template Included)

When I say “write it like unit tests,” I mean: define pass/fail conditions you can point to.

Here’s a copyable LLM evaluation rubric template I’ve used (lightweight, but solid):

Rubric

Intent 

“Answer question using RAG context”

• Correctness (0/1/2)

  • Answer matches facts in context and addresses the user’s question.

  • Partially correct or incomplete, but no major factual errors.

  • Wrong or ignores the question.

• Groundedness (0/1/2)

  • Claims are supported by provided context; no unsupported specifics.

  • Mostly grounded but includes a small unsupported claim.

  • Hallucinates key facts or references nonexistent sources.

• Use of Context (0/1)

  • Uses the relevant retrieved info (correct doc/section).

  • Misses obvious relevant context or uses irrelevant chunks.

• Handling Uncertainty (0/1)

  • If context is insufficient, says what’s missing and asks a precise question or recommends next step.

  • Confidently guesses or hand-waves.

• Format/Policy (0/1)

  • Follows required format and policy constraints (tone, refusal rules, etc.).

  • Violates format/policy.

depends on severity; Sev 0 often requires no 0s.

That’s already enough structure to run a real release gate.

Handle Ambiguity Without Endless Debate

Ambiguity is normal in production. Users are vague. Context is missing. Policies conflict. Your job is to define what “good” looks like when the world is messy.

My rule of thumb:

• If the user’s intent is unclear, a “good” answer asks a clarifying question or provides bounded options “If you mean X, do Y; if you mean Z, do W”

• If the context is insufficient, a “good” answer does not fabricate. It should say what it needs.

So your rubric should explicitly allow “ask a clarifying question” as a valid expected behavior. Otherwise labelers will punish the model for not mind-reading.

Keep Labels Reliable Over Time

Rubrics drift when:

• new people join,

• product goals change,

• the model’s behavior shifts,

• edge cases become common cases.

A few practical habits help a lot:

• Double-label 10–20% of cases each week (lightweight agreement check).

• Keep a tiny “rubric examples” page  2–3 examples of pass/fail per intent.

• When a disagreement happens, update the rubric with a concrete example. Don’t just “agree to disagree.”

You’re not trying to be statistically perfect. You’re trying to avoid accidental chaos.

Can You Use LLM-as-a-Judge?

Yes, you can use an LLM to grade outputs. And yes, it can save you time. And yes, it can also lie to you with incredible confidence.

Where it works well:

• Format checks (did it return JSON? did it include citations?)

• Simple rubric items (“did it mention the refund policy?”)

• Pairwise comparisons (A vs B) when you’re choosing between prompts

Where it bites people:

• Subtle correctness (it may “agree” with a wrong answer)

• Groundedness (it might not notice hallucinated specifics)

• Bias toward longer, more fluent answers

How to calibrate:

• Start by having humans label 50–100 examples.

• Run LLM-judge on the same examples.

• Measure agreement, but more importantly: inspect the disagreements.

• Update the judge prompt with your rubric and include 3–5 labeled examples (few-shot) of what “pass” and “fail” look like.

And keep one hard rule: for Sev 0 cases, trust but verify. I’ve seen teams ship a safety regression because the judge model was “too polite” to fail outputs that sounded reasonable.

Metrics That Matter Beyond Accuracy

If you reduce everything to “accuracy,” you’ll make bad product decisions. LLM apps fail in multiple ways that don’t show up in a single number.

Here are the metrics I actually care about.

Task Success

This is the most product-shaped metric: “Did the user get what they needed?”

Task success can be:

• binary (completed / not completed),

• graded (0/1/2),

• or workflow-based (“did they reach the next step without human intervention?”).

For example, in a support bot:

• Success might mean the user got the correct policy + next action + link to the right page.

• Failure might mean incorrect policy, or correct policy but no actionable step, or a tool call that didn’t happen.

Task success is usually rubric-driven. That’s okay. It’s still better than vibes.

Groundedness and Hallucination Resistance

If you do RAG evaluation, you care deeply about groundedness: the model’s ability to stay within provided sources.

Useful metrics:

• Unsupported claim rate

  percentage of answers containing claims not supported by the given context.

• Citation precision

  when it cites, are the cited chunks actually relevant?

• Refusal/uncertainty correctness

  does it admit when context is missing?

This is where “bad labels” happen a lot, because people write expected answers like “Should mention X” without specifying whether X exists in the context.

Here’s what a bad label looks like:

• Expected

  “Answer with the current pricing tiers.”

• Context

  does not include pricing tiers.

• Result

  labelers punish a model that correctly says “I don’t have pricing info,” and reward one that hallucinates tiers.

That’s how you train your team to prefer confident nonsense.

Retrieval/Tool Metrics

For tool-using agents and RAG systems, you should measure the pipeline, not just the final text.

Examples:

• Retrieval recall proxy

  “Did we retrieve the doc that contains the answer?” (Often measured by whether a known relevant doc ID appears in top-k.)

• Top-k hit rate

  by intent bucket.

• Tool correctness

  did it call the right tool, with the right parameters, at the right time?

• Tool efficiency

  number of tool calls per successful task (cost + latency).

In practice, a lot of “model regressions” are actually retriever changes, chunking bugs, or tool schema mismatches. If you don’t track these, you’ll blame the wrong thing and waste a week.

Safety and Risk Metrics

Even if you’re not building a “safety product,” you still have safety risks:

• leaking private data,

• giving medical/legal advice,

• generating policy-violating content,

• prompt injection in RAG

Metrics I like:

• Refusal accuracy on disallowed requests.

• Sensitive data leakage checks (does it repeat secrets from context?).

• Injection resistance (does it follow malicious instructions inside retrieved docs?).

For high-risk apps, include Sev 0 safety cases in your Smoke set. Run them constantly. Don’t “get to it later.”

Operational Metrics

This is the stuff that gets you paged:

• latency

• cost per request,

• token usage,

• tool failure rate/timeouts,

• “empty answer” rate,

• retry rate.

A model that’s slightly more accurate but doubles latency might be a net loss. Your eval dashboard should make that tradeoff visible.

Report Metrics by Slice

“One average score” hides the truth.

Slice your metrics by:

• intent bucket,

• severity,

• language,

• customer tier,

• context length,

• “requires RAG” vs “no RAG,”

• tool vs non-tool flows.

You’ll often discover you improved the average but regressed the high-value slice. And then you get to explain that to a customer. Ask me how I know.

A Minimal “Golden Set” Template You Can Copy

You don’t need a complicated system. You need a consistent schema.

Common Pitfalls

Too big too early

If you start with 500 cases, you’ll never finish labeling, and your pipeline won’t exist.

Do this instead

ship a 20–40 case Smoke set first. Make it runnable. Then expand.

No rubric examples

Teams write rubrics like “be helpful,” then argue for days about what that means.

Do this instead

include 2–3 “this passes” and “this fails” examples per intent bucket.

Missing context

People save but not the retrieved chunks or tool output. Then results become non-reproducible.

Do this instead

store the exact context used for the test case, or snapshot it (doc version + chunk IDs).

One average metric

An average score hides regressions in high-severity cases.

Do this instead

report by severity + intent slice. Make Sev 0 failures impossible to miss.

No versioning

If you can’t answer “when did this change,” you’re doing archaeology every incident.

Do this instead

version the golden set and store run metadata (model version, prompt hash, retriever config).

Ignoring production drift

User behavior changes. Docs change. Tools change. Your eval set goes stale.

Do this instead

add a weekly refresh habit: pull recent logs, add 5–10 new cases, retire outdated ones.

A quick war story

We had a support assistant doing RAG over internal docs. Someone “improved chunking” to reduce token usage (reasonable goal). The assistant still sounded great very confident, very fluent. But the retriever started missing a specific policy paragraph that only appeared in one doc variant.

No one noticed in casual testing. The average score looked fine.

Two days later, a big customer got the wrong policy answer, escalated, and we had a mess. The painful part: the model didn’t hallucinate randomly. It answered plausible-sounding policy based on incomplete context.

The fix wasn’t “better prompting.”

It was:

• adding a Sev 0 golden case for that policy question,

• tracking retrieval hit rate for the relevant doc,

• and alerting when that hit rate dropped.

That incident paid for the eval pipeline in one week.

What to Do Next: Evals as a Weekly Habit

Evals aren’t a one-time project. They’re a habit that keeps your team sane.

A weekly cadence that works:

• Every week

  add 5–10 new real cases (from recent logs, escalations, new features).

• Every week

  double-label a small slice (10–20%) to catch rubric drift.

• Every release

  run Smoke + Release sets, block on Sev 0 failures.

• Every sprint

  review the worst slices (intent bucket + severity) and decide what to improve next.

Checklist I use:

• Did we add any new user intent? Add cases.

• Did policies/tools/docs change? Update context snapshots.

• Did we have any incidents/escalations? Add them to “Hall of Shame.”

• Did we change model/prompt/retriever? Run regression tests and store results.

---

You Might Be Interested In

• What Are Open Source Alternatives To Cuda?
• How To Use Ai Tools Without Leaking Sensitive Company Data?
• How Does A Software Automation Platform Streamline Tasks?
• 7 Ai Infographic Tools That Impress
• Can I Study Ai For Free?

---

Conclusion

If you take one thing from all of this: an evaluation set for an LLM app isn’t a research artifact it’s your shipping safety net. The best teams I’ve seen don’t “do evals” as a one-off push before launch. They build a small golden set, wire it into their release flow, and treat regressions like real bugs.

The practical recipe is boring on purpose:

• Start tiny (20–40 Smoke cases) so you actually finish.

• Capture the full situation (inputs + context + tool outputs), not just the user prompt.

• Score with a checklist rubric, not vibes.

• Track severity and slices so you don’t hide Sev 0 failures behind a nice average.

• Version everything so you can answer “when did this change?” without archaeology.

• Refresh weekly so your set doesn’t drift while production does.

You don’t need months. You need two focused weeks to get a runnable baseline, then a weekly habit to keep it alive. Once that’s in place, you can move fast and know what you’re breaking before your users do.

FAQs

What is a golden set for an LLM app?

A golden set is the small, high-signal collection of test cases you trust enough to run all the time. It’s “golden” because it represents the workflows you really care about (the stuff tied to revenue, retention, or safety), plus the failures that have bitten you before. In practice, it’s not a huge dataset and it’s not meant to be exhaustive. It’s the set you’d use to answer, with a straight face, “Did this change break anything important?”

The key is that each case includes the context that actually drives behavior retrieved chunks for RAG, tool outputs for agents, and the conversation history for multi-turn flows plus a rubric that defines what “good” means. Without that, you don’t have a golden set; you have a pile of prompts that produce random-looking results every run.

How many examples do I need?

You need fewer than you think to get real value. For most teams, the first milestone is a Smoke set of roughly 20–40 cases: enough to cover your top intents and the top ways things fail, not enough to become a labeling death march. If you can run that set quickly and repeatedly, you can catch obvious regressions and stop shipping blind. That’s already a massive upgrade over “we tested a few prompts in a notebook.”

After that, you grow into a Release set (often 80–200) that has broader coverage and more edge cases, and only later a Deep set (300–1,000+) if you’re doing systematic improvements and want stable, sliceable metrics. The sizing “depends” mainly on how many distinct behaviors your product has (RAG + tools + multi-turn adds complexity fast) and how expensive a regression is. High-risk or high-revenue apps should bias toward more coverage sooner but still start small enough to actually finish.

How do I label consistently?

Consistent labeling comes from refusing to label based on vibes. If two smart people can read the same output and disagree, that’s not a “people problem,” it’s a rubric problem. The fix is to turn “good answer” into observable checks: did it answer the question, did it use the provided context, did it fabricate anything, did it follow format/policy, did it ask a clarifying question when needed. Once you write those down as a checklist (and include a couple of pass/fail examples), labeling gets dramatically less subjective.

The other half is being disciplined about context. A lot of “inconsistency” is just missing context: labelers imagine different retrieved docs, different tool outputs, or different earlier turns. Snapshot what the model saw (or at least the identifiers and versions), and make “context insufficient → ask a clarifying question / say what’s missing” an explicitly acceptable outcome. That prevents the classic failure mode where you accidentally reward hallucinations because they sound confident.

What metrics matter beyond accuracy?

Accuracy is a blunt instrument for LLM apps because the failure modes aren’t just “right vs wrong.” In real systems you care about task success (did the user actually get to the next step), groundedness (did it stay faithful to the provided context in RAG), and behavior under uncertainty (did it admit what it doesn’t know instead of guessing). A model can be “accurate” on easy cases while still being dangerous or expensive on the cases that matter like confidently inventing policy details when the retriever misses a chunk.

You also want pipeline metrics that tell you why you’re failing. For RAG, that’s retrieval hit-rate proxies and citation precision; for agents, tool-call correctness and tool efficiency (too many calls means latency and cost blow up). And don’t ignore operational metrics like p95 latency and cost per request those are product constraints, not afterthoughts. The most important reporting trick is slicing: a single average hides regressions in your highest-severity workflows, which is exactly where you can least afford surprises.

Can I use an LLM to grade?

Yes, and it can be a big accelerator but only if you treat it like a tool you calibrate, not an oracle you blindly trust. LLM-as-a-judge works best for rubric items that are clearly checkable (format compliance, presence/absence of required elements, basic groundedness checks when context is explicit) and for pairwise comparisons when you’re choosing between two prompts or two models. It’s especially handy when you need fast iteration and you can’t afford human review on every run.

Where teams get burned is assuming the judge is “objective.” Judges can be biased toward fluent, longer answers, and they can miss subtle factual errors or hallucinations sometimes the most damaging kind. The practical approach is to start with a human-labeled calibration set (even 50–100 cases helps), measure where the judge disagrees, and then tighten the judge prompt with your rubric and a few labeled examples. For high-severity cases (safety, policy, anything that can cause real harm or major customer fallout), I still keep at least some human review in the loop, because a confident judge model can fail in exactly the same way as the model you’re evaluating.