Pick two metrics per use case, not ten. One should track the business outcome, one should act as a guardrail against regressions, and both should run against a frozen test set held in version control. Everything else is diagnostics for the engineers, useful during debugging and worthless in a steering committee.
Most evaluation work fails for a dull reason: the test set. Teams argue for six weeks about which grader model to use, then run it over 40 examples copied out of a demo. The choice of metric matters far less than whether the data underneath it resembles what users actually send.
The four metric families
| Family | Typical metrics | How it is computed | Where it breaks |
|---|---|---|---|
| Deterministic | exact match, F1, SQL execution accuracy | string or execution comparison against a reference answer | tasks with many correct answers |
| Retrieval | recall@k, precision@k, hit rate | labelled query-document pairs | labels generated by the same pipeline under test |
| Model-graded | faithfulness, answer relevance, instruction adherence | a second model scores output against a rubric | vague rubrics, or a grader sharing the generator’s blind spots |
| Production | thumbs-down rate, retry rate, human escalation, task completion | application telemetry | low traffic, or users who leave rather than complain |
Push as much of your task as you can into the deterministic row. A support assistant that must cite a policy ID can be scored on whether the right ID appears. A classifier wrapped in a chat interface is still a classifier. Model-graded metrics are the fallback for genuinely open-ended output, and they cost real money: 500 test cases, graded on every pull request, is a recurring inference bill that teams forget to budget.
What an evaluation suite costs to build
A first suite covering one use case runs 8 to 15 engineer-days. The split is predictable:
- Test set construction and labelling: 3 to 6 days for 200 to 500 cases. This is the largest line and the one every team underestimates.
- Runner and CI integration, so a failed evaluation blocks a merge: 2 to 4 days.
- Rubric design and calibration of the grader against human labels: 2 to 3 days.
- Reporting that a non-engineer can read: 1 to 2 days.
Multiply by whatever you pay per day; current European senior ranges sit on our daily rates page. Then add 1 to 2 days per month per use case for maintenance. The test set goes stale as the product changes, and a suite nobody updates is worse than none, because it reports green on last quarter’s problem.
Put the metric in the contract
Name the metric, the test set version and the threshold in the statement of work. “Faithfulness above 0.85 on test set v3, measured with a pinned grader model” is a condition you can argue about. “High quality responses” is not. If a vendor resists a numeric acceptance criterion at the point of signing, you will be relitigating scope at the point of delivery.
The team shipping the feature should own evaluation. Handing it to a separate QA function produces a scoreboard nobody acts on, and the same ownership logic that governs MLOps applies.
Where the scores mislead
- Averages hide catastrophic failures. A 0.91 mean with a 4% tail of confidently wrong answers is a worse product than a 0.85 mean with no tail. Report the worst bucket separately.
- Generation metrics absolve bad retrieval. Faithfulness only asks whether the answer follows the retrieved context. If the context was wrong, the score is high and the user is misinformed, so score retrieval separately from generation.
- Grader model versions move. Pin them, and re-baseline deliberately rather than waking up to a two-point shift you cannot explain.
- On 200 cases, a two-point improvement is noise. Either enlarge the test set or stop celebrating small deltas.
When renting engineers for this is wrong
If your bottleneck is domain judgement rather than engineering, an external engineer will not fix it. A test set for clinical triage or contract review needs several hundred labels from people holding the professional qualification, and that time comes from your own staff. We can build the evaluation suite around their labels. We cannot produce the labels.
Two further cases where you should not call us. If the feature has no production traffic, build it and collect 500 real queries first. And if you have one use case and an in-house team with two spare weeks, 10 days of work does not justify a vendor contract. Come back when it is five use cases on a shared platform, the point at which senior AI engineers on a time-and-materials basis start to pay for themselves.