Evals for Marketing AI: Numbers Instead of Vibes

Ask a team whether their AI output improved after last week's prompt change and you get a confident answer: it feels tighter. Ask which of last month's outputs would fail today and the room goes quiet, because nobody kept them and nobody wrote down what failing means.

Impressions are a real signal and a terrible instrument. They arrive late, they cannot be compared across weeks, and they are strongest in whoever made the change.

The golden set is the asset, and the dashboard is not

An eval set is a fixed collection of inputs with known-good expectations, run identically on every change. Building it is the work; everything after is arithmetic.

Take the examples from production, not from imagination. Sample real requests from your logs across the traffic you serve, then add the hard cases the people doing the job by hand know about. Do not build the set from the examples you used to develop the prompt: those are what it was tuned to pass, and they will tell you everything is fine.

Thirty to fifty cases per task is enough to start and to catch a gross regression. A hundred and fifty to three hundred shows movement of a few percentage points. Below about thirty, one flipped case swings the score by more than the effect you are looking for.

Every failure you have already seen belongs in the set

This is the rule teams skip, and the one that compounds. When something goes wrong in production, the fix is not complete until the case is in the eval set with the correct output written down. Otherwise the same defect returns in six months wearing a different prompt.

That turns the set into a regression suite rather than a benchmark, and grows it along the shape of your actual weaknesses instead of somebody's idea of coverage.

Run this check in ten minutes. Open the last three AI-related complaints your team received and see whether any exists as a test case. In most organisations the answer is none.

Different tasks need different numbers

One score for "quality" is how teams end up arguing. Match the metric to the task.

Classification. Per-class precision and recall, with F1 where you need one number. Accuracy alone lies on imbalanced classes, and the confusion matrix matters because the direction of the error has different costs: a lead wrongly routed to enterprise sales is not the same mistake as one silently discarded.

Extraction. Exact match on fields where a near miss is still wrong: dates, amounts, identifiers. F1 over sets of extracted items where partial credit is meaningful. And a schema validity rate, because output that does not parse is a failure whatever it says.

Retrieval. Measure it separately from generation, as the retrieval pipeline argument sets out. Hit rate at five says whether the correct passage reached the model. Citation correctness, whether the cited span supports the sentence, and groundedness, the share of claims supported by context, say what the model did with it. Track abstention too.

Generation. A rubric of three to five dimensions on a short scale with written anchors, plus as many binary checks as you can write. Binary checks are reproducible between two readers. A seven-point "clarity" scale is not.

A model judging a model has four specific weaknesses

Using a model to score outputs is the only way to evaluate generation at volume. Its biases are documented, predictable and controllable once you know them.

Position bias. Given two candidates, judges favour one position over the other. Run every pairwise comparison in both orders and treat disagreement as a tie.

Verbosity bias. Longer answers score higher, correctness being equal. Compare at controlled length, or make brevity an explicit rubric dimension so the effect is visible rather than hidden.

Self-preference. Models rate text from their own family more generously, so where the comparison decides something, use a judge from another family than the generator.

False precision. Judges are poor at fine-grained scoring and much better at comparison. Prefer a pairwise "which is better" or a three-point rubric over a score out of ten, which produces stable-looking numbers that mean little.

Calibrate the judge before you trust it

Label sixty to a hundred outputs by hand against the same rubric, with two people where the judgement is contested. Run the judge on those outputs and measure how often they agree. Percentage agreement is enough to start; Cohen's kappa is better because it discounts chance agreement.

If the judge disagrees with your reviewers on a fifth of cases, its score is decoration and the rubric is what to fix. Re-calibrate whenever the judge model, the rubric or the task changes, and keep the human-labelled set so calibration can be repeated.

A threshold that blocks a merge is what makes it an eval

A score nobody acts on is a metric. An eval is a gate. Run the set automatically on every change to a prompt, a model route, a retrieval setting or a context file, under two rules: the aggregate must not fall below the baseline, and named critical cases must never fail.

Full runs cost tokens and minutes, so use two tiers: a fast subset of thirty cases on every pull request, the full set nightly and before release. Report the case-level diff rather than the headline number, because "84% to 82%" tells a reviewer nothing while "these four cases flipped" is actionable. Regressions can be accepted, but only in writing, the way prompt review already works.

Providers change models underneath you

A model name is not a promise of stable behaviour. Providers update, retire and re-tune, and output can move with no change on your side. If you first hear about it from a customer, the eval set was only measuring your own edits.

So run the set on a schedule as well as on changes, weekly as a default, and keep the score history with the model and prompt version against each run. That history separates your regression from theirs in the twenty minutes after somebody notices. The routing layer holds the version pinning that makes the comparison valid.

What an eval set cannot see

It measures what you thought to include. The failure nobody imagined is invisible to it, and confidently wrong output that reads well is exactly that kind of failure. Only sampled human review of production output catches it, and late.

It goes stale. The product changes, the corpus changes, the audience changes, and cases written a year ago test behaviour nobody wants. Review the set quarterly and retire what no longer represents anything real.

And a passing score is not a good result. Sets drift toward being easy, and teams optimise toward the measure until it stops standing for the thing. Keep a rotating holdout of recent production samples nobody tunes against, and treat the eval as evidence that quality has not fallen rather than proof it is high. That is the difference between a measured AI system and a well-instrumented one.

Frequently asked questions

How many examples does an AI eval set need?

Thirty to fifty per task is enough to begin and to catch obvious regressions. A hundred and fifty to three hundred lets you detect movement of a few percentage points. Below roughly thirty, one case flipping moves the score more than the change you are measuring. Start small with real production examples rather than waiting to build something comprehensive, and grow the set every time something fails.

What metrics should I use to evaluate AI output?

Match the metric to the task. Classification wants per-class precision and recall with a confusion matrix, since accuracy hides imbalance. Extraction wants exact match where a near miss is wrong, F1 where partial credit makes sense, and a schema validity rate. Retrieval wants hit rate at k measured separately from the answer, plus citation correctness and groundedness. Generation wants a short rubric with written anchors and binary checks.

Can you use an LLM to judge AI output quality?

Yes, at volume it is the only practical option, but calibrate it first. Model judges show position bias, favour longer answers, rate their own family more generously, and produce unreliable fine-grained scores. Mitigate by running comparisons in both orders, controlling for length, using a judge from a different family than the generator, and preferring pairwise comparison to a score out of ten. Then check agreement against human-labelled examples.

How do you catch quality drift when a provider changes a model?

Run the eval set on a schedule rather than only when you change something, weekly for anything important, and store every run with the model and prompt version attached. Behaviour can move with no edit on your side, so a scheduled baseline is the only way to separate a provider's change from your own. Version pinning at the gateway makes the comparison valid, and the history turns a vague suspicion into a dated regression.