A team tries five questions, likes the answers and decides the agent is ready. Those examples may prove that the demonstration can work. They say little about the request with missing information, the customer who asks for a person or the tool call that changes the wrong record.
An evaluation set makes those expectations explicit. It gives the business and implementation team a shared way to decide whether a change improves the system or introduces an unacceptable failure.
It is not a certification scheme or a claim that passing a small test set proves universal safety or reliability.
Start with the job the agent must finish
Describe the workflow in business terms. For example, an internal assistant helps an account manager find the current onboarding procedure. A support agent answers a documented setup question and routes unsupported requests. A scheduling agent records the agreed appointment.
Define what a good outcome looks like for each. Some cases should produce an answer; others should produce a clarification, refusal or human handoff. If the rubric rewards only fluent answers, it can penalize the behavior you actually need.
NIST's AI Risk Management Framework overview, reviewed October 27, 2026, describes voluntary guidance for incorporating trustworthiness into AI design, use and evaluation. The page notes that AI RMF 1.0 is being revised. The practical rubric below is VUZURAN editorial analysis, not a claim of NIST approval or compliance.
Create a small set with meaningful variation
Begin with cases drawn from the actual workflow, using authorized or synthetic data. Include ordinary examples and the conditions that could cause harm or unnecessary work. Record why each case belongs in the set.
Do not count ten rewordings of the same easy request as broad coverage. Vary the evidence, permissions, ambiguity and external-system behavior. A shorter set with distinct failure modes can reveal more than a larger collection of near-duplicates.
Keep the expected result precise enough for two reviewers to compare. “Good response” is too vague. “Identifies the current document, cites the relevant section and does not reveal the restricted contract” is reviewable.
Use examples that include stopping correctly
The following cases are hypothetical. Adapt them to your workflow and source material rather than copying the expected behavior without checking your own policies.
| Test case | Expected behavior | Failure to record |
|---|---|---|
| Approved source answers the question | Give a supported answer with a useful reference | Citation does not support the claim |
| Required information is absent | State the gap or ask a focused question | Invented value fills the blank |
| User lacks access to a document | Respect the access boundary | Restricted information appears in the answer |
| Two records share a name | Request or use a stable identifier | Action targets the wrong record |
| Customer asks for a person | Follow the defined handoff | Repeated automated loop blocks the request |
| Remote action times out | Preserve uncertainty and reconcile | Blind retry creates a duplicate |
A correct decision to stop can be a successful evaluation result.
Add at least one case where an apparently relevant source is outdated. The agent should not treat a familiar document title as proof that the content still governs the work.
Separate hard requirements from quality preferences
Permission failures, wrong-recipient messages and unsupported external actions should not disappear inside an average writing score. Define release-blocking conditions separately from preferences such as tone or brevity.
For each case, score the relevant dimensions: task correctness, evidence support, boundary handling and clarity. Not every dimension applies equally to every task. A deterministic field extraction may need exact validation, while a summary may allow several acceptable phrasings.
Write examples of acceptable and unacceptable outcomes for reviewers. This reduces disagreements caused by different interpretations of the rubric rather than differences in the system's actual behavior.
Preserve a final test set
Use one set to develop the workflow and a separate set for the release decision. If every failure is fixed by adding its exact answer to the prompt, the team may learn little about behavior on new requests.
Keep configuration details with the evaluation record: prompt version, model or route, tool setup and source collection. Without that context, a later team cannot reproduce the result or understand what changed.
Where outputs vary, repeat appropriate cases and record the variation. Do not select only the best response from several attempts and report it as the system's ordinary performance.
Editorial principle: the evaluation should describe the system people will use, including its inconvenient cases.
Include the surrounding application
A language-model response is only one part of an agent workflow. Test retrieval permissions, tool validation, approval handling and destination confirmation. A strong answer can still lead to a wrong action if the application passes the wrong record identifier.
Run controlled integration tests in an authorized environment. For an appointment workflow, inspect the actual date, timezone and calendar record. For a CRM task, verify the intended owner and duplicate behavior. Our model-routing guide explains why the router also needs its own evaluation.
Do not replace these checks with a model grading its own explanation. Automated judging can help triage some outputs, but consequential cases need evidence and an accountable review process.
Turn failures into specific changes
Group failures by cause. Missing source material requires a different fix from incorrect retrieval. A permission bug requires a different response from an overly long answer. A destination timeout may require recovery logic rather than another prompt revision.
After a change, re-run the relevant cases and the earlier accepted set. Record improvements and regressions. Keep the release decision tied to the defined requirements instead of moving the standard to fit whichever results look best.
Use production incidents to expand the set after appropriate data review. A newly discovered edge case is valuable evidence, especially when the original tests did not represent that part of the workload.
Agree on who accepts the release
The software team can establish that tests ran and report the evidence. The business owner must confirm that the behavior meets the workflow's requirements. Assign ownership before the final review so uncertain cases do not become an informal debate after launch.
VUZURAN develops custom AI tools and workflows. A project discussion can begin with a small acceptance set and the consequences of failure. That gives the team a more concrete brief than a request for an agent that simply “works well.”
Frequently asked questions
How many evaluation cases do we need?
There is no universal count. Cover the actual task types, important boundaries and known failure modes, then expand as evidence reveals gaps.
Can an AI model grade the results?
It can assist with some review tasks, but its judgments need validation. Do not use a model score as the sole proof of permissions, external actions or consequential correctness.
Should a refusal count as a failure?
Only if the task should have been completed. When evidence or authority is missing, stopping or asking for clarification may be the correct outcome.
When should we re-run the set?
Re-test after relevant prompt, model, source, tool or workflow changes. Preserve the configuration and compare against the previous accepted behavior.
Sources & context
- NIST: AI Risk Management Framework
Primary documentation reviewed October 27, 2026. Original editorial frameworks and explicitly hypothetical examples; not provider-endorsed or measured customer results.
Written by VUZURAN. Examples labeled hypothetical are illustrative; they are not customer results. VUZURAN service references describe our own business.