Evaluating an AI feature without relying on vibes

Someone changes a word in a system prompt. Everything looks fine in the three examples they tried. Two weeks later support notices the feature has been quietly worse since that deploy, and nobody can say which change did it.

This happens because the team has tests for the code around the model and nothing for the model’s behaviour. The fix is an eval set, and it is much less work than it sounds.

Start with fifty real inputs

Not synthetic ones. Pull actual production inputs — sampled across the range you really see, not the tidy ones.

Fifty to a hundred cases is enough to catch regressions. People delay building this because they are imagining thousands; you do not need thousands to notice that a prompt change broke JSON output on every input containing a quotation mark.

Weight it deliberately:

• Every failure you have already had. This is the highest-value part of the set and it costs nothing to collect: every time someone reports a bad output, the input goes in the file. An eval set that grows one incident at a time becomes a regression suite for free.
• Boring, typical cases. If a change breaks the common path you want to know immediately.
• The edges you know about. Empty input, enormous input, wrong language, adversarial phrasing, the customer whose name contains an apostrophe.

Store it as data next to the code, in version control. A JSONL file with an input and whatever you can assert about the output is enough to start.

Grade with code wherever you can

The instinct is to reach for a model to judge quality. Resist it for anything a program can decide.

Deterministic checks cost nothing, never disagree with themselves, and cover more than people expect:

• Does it parse as valid JSON, and match the schema?
• Is the classification label from the allowed set?
• For extraction, does the extracted value match the known answer exactly?
• Are required fields present, and is anything present that should not be?
• Did it stay under the length limit?
• Does the generated command run, or the generated manifest validate?

That last one is the strongest form available: for code and configuration generation, the check is whether the output works. kubeconform on a generated manifest is a better judge than any model.

Structured output support helps here, because constraining the response to a schema turns a whole category of format failures into a non-issue rather than something to test for.

LLM-as-judge, and its limits

For open-ended output — a summary, an explanation, a postmortem draft — there is nothing to assert exactly, and a model grading against a rubric is the practical option. It works, with caveats worth knowing before you trust the number.

Absolute scores drift; comparisons hold up. “Rate this summary 1–10” produces numbers that move when you rephrase the rubric. “Here are two summaries, which better captures the timeline?” is far more stable. Prefer pairwise comparison against a reference output.

Position bias is real. Judges favour whichever response came first. Run each comparison both ways round and count a disagreement as a tie.

A judge cannot check facts it does not have. It will happily approve a fluent summary containing an invented detail. If factual accuracy matters, give the judge the source material and ask it to identify claims not supported by it — that is a checkable task, unlike a quality score.

Never judge with the model being tested, on its own output. Use a different model, or at minimum a separate call with no knowledge of which system produced what.

Calibrate the judge once against human ratings on twenty cases. If it disagrees with your team more than your team disagrees with itself, the judge is not usable and the rubric needs work.

Running it in CI without flakiness

Model outputs vary, so a naive assertion produces a suite that fails randomly and gets disabled within a month.

Three practices keep it useful:

Gate on aggregate, not per-case. Fail the build when the pass rate drops below a threshold, not when any single case flips. Track the rate over time.

Pin what you can. Pin the model version explicitly. A silent upgrade underneath you is indistinguishable from a regression you caused, and you will spend a day proving it.

Run on the changes that matter. Every prompt change, every model change, every retrieval change. Not on every commit — evals cost money, and a README edit cannot move the number.

Report the diff, not just the total. “Pass rate 87%” is much less useful than “these four cases regressed, they were passing yesterday”. The regressed cases are the output; the percentage is a summary of it.

Batch processing suits this well: an eval run is not user-facing and tolerates an asynchronous window at roughly half the cost. We covered the economics in cutting LLM cost in a pipeline.

What to measure besides quality

Quality is one axis and rarely the one that causes an incident. Track alongside it:

Signal	Why it matters
p95 latency	Where the user experience actually breaks
Cost per call	Catches a prompt that quietly tripled in size
Refusal rate	A sudden rise means the prompt or inputs changed
Format failure rate	Cheap to fix, expensive to leave
Cache hit rate	Falls silently when a prompt prefix changes

A dashboard with those five and a pass rate tells you more about the health of an LLM feature than any amount of trying it by hand.

The point of all this

An eval set does not prove the feature is good. It proves it has not got worse since the last time somebody checked, which is the property you need in order to change anything with confidence.

Without one, every prompt change is a gamble nobody can settle, and teams respond by not changing prompts at all. That is the real cost of skipping it: not bad output, but a system that ossifies because no one dares touch it.