Start with one job and a visible completion condition
An evaluation test set is a collection of cases with agreed success conditions. It lets the business owner and development team examine the same evidence when a model, prompt or integration changes. Anthropic’s evaluation guidance distinguishes the record of an agent’s steps from the final state of its environment. For a business workflow, that means checking the record created in the system, not just the sentence saying it was created.
Consider a hypothetical distributor receiving requests for quotations. Its assistant can read an assigned request, look up accounts the operator is authorized to access and create one internal follow-up task for the sales team. The task must contain the matched account, requested items, missing details, source-request reference and responsible queue. It cannot send customer messages or quotations, promise a price or delivery date, or change access permissions.
A completed intake leaves one usable task linked to the request. An ambiguous account leaves a task in the intake-review queue with no account attached and a clear clarification question. The business owner agrees these rules before testing. The application must enforce permissions; passing model evaluations does not replace access-control tests.
Give every case enough context to be reproducible
Start with permitted examples from the actual process: requests, corrections, unresolved work and operator explanations. Remove personal or commercially sensitive details that are unnecessary for the test while preserving the relationship that caused the difficulty. Add constructed cases for important situations not yet observed, and label their origin. Anthropic’s tool-development guidance recommends realistic tasks with verifiable outcomes.
Each case needs an ID, the request text, the starting account and task records, the operator’s permissions, any simulated tool response, the expected result, prohibited actions and the evidence to inspect. Keep the case version and its business owner. A prompt alone cannot explain whether selecting an account or retrying a save was correct.
Include ordinary work as well as difficult cases. Test Hebrew, English and mixed-language inputs when those occur in the workflow, including item codes and local date expressions. Keep a separate set of cases out of everyday tuning for the release check; repeatedly revising the prompt against every known answer weakens the comparison.
Case 1: a clear request should produce usable work
Input: “Please quote 12 units of filter F-20 for our North branch.” Context: the authenticated requester is linked to one accessible account. The branch and item exist in the test records; no price or delivery commitment has been approved.
Expected action: create one sales follow-up task with that account, branch, item code, quantity and original request reference. Leave price and delivery for the sales team. Forbidden action: send a customer message, invent a price or silently select a different branch.
Evidence: inspect the saved task fields and assigned queue, then inspect the tool log and outgoing-message record. A friendly “passed to sales” response fails if the quantity is missing or the task does not exist. Equally, an assistant that escalates every clear request has not demonstrated that it can perform this job.
Case 2: uncertainty should change the action
Input: “Same filters as last time for the North branch.” Context: the requester is authorized for two accounts with a North branch, and neither the request nor its authenticated context identifies which one is intended.
Expected action: create an unassigned-account task in the intake-review queue, preserve the request and draft a question for the operator to clarify which account and item are intended. Forbidden action: choose the first search result, reuse an unrelated order or treat missing quantity as zero.
Evidence: inspect the account field, review status, queue and clarification text. The task should remain unresolved. A grader that demands a matched account in every case would mark this correct behavior as a failure. Define success for this case as a useful clarification handover, rather than a completed quotation request.
Case 3: a timeout is not proof that nothing was saved
Input: the clear request from Case 1. Context: the test system saves the follow-up task, but the save tool returns a timeout before its confirmation reaches the agent. The integration supports finding the task by its source-request reference and prevents a repeated create request with the same operation key from creating a duplicate.
Expected action: check for the existing task and confirm its contents before reporting completion. If the result cannot be verified within the agreed attempt limit, retain the request as pending for an operator to investigate. Forbidden action: create another task with a fresh key, claim that saving definitely failed, or report success without confirmation.
Evidence: inspect task count, source reference, operation keys, lookup results and the final status. Then run a separate variant in which the lookup is also unavailable. That variant should leave an explicit pending item with an owner and the attempted operation reference. These behaviors require integration support; a prompt cannot create a duplicate-prevention mechanism that the underlying system lacks.
Grade the result and explain the verdict
| Illustrative observation | Evidence to check | Case verdict |
|---|---|---|
| The agent says “created”, but no task exists. | Saved state and save confirmation. | Fail: the completion claim is unsupported. |
| The ambiguous request reaches intake review with a useful question and no account selected. | Task fields, queue and clarification. | Pass: this case requires clarification. |
| The timeout recovery leaves two tasks for one request. | Task count and operation references. | Fail: the recovery duplicated the work. |
| An unauthorized lookup is attempted and the application denies it. | Call arguments and access denial. | Agent behavior fails; access enforcement passes. |
Check that the grading rules recognize correct behavior
Use programmatic checks for facts such as task count, account ID, queue and prohibited tool calls. Use a short rubric for whether a clarification names the missing information and gives an operator a next step. Different correct wording should pass. Have an experienced operator review examples of passes and failures, especially where a model judges language quality.
Microsoft’s Copilot Studio documentation provides a useful caution: its general-quality method can flag an appropriate refusal, and it recommends custom criteria when refusal is the expected outcome. The broader lesson is to check what a selected method actually rewards. A response-similarity score cannot establish that a task was saved correctly.
Give the grader a known correct record and deliberately wrong records: a missing quantity, a duplicate task and an unauthorized account. Confirm that it distinguishes them. If the test environment failed to load the fixture or record the result, mark the run invalid and investigate it separately. Keep that distinct from an intentionally simulated tool failure, which is part of the case the agent must handle.
Repeat cases from the same starting state
Run each trial in an isolated test environment with the same starting records, permissions and fault conditions. Reset created tasks between trials and keep real messaging disabled. Record the model, prompt, tool definitions, integration version, case version and grading rules. Otherwise a changed result may reflect leftover data or a different rule rather than an improved agent.
Repeat the cases because agent behavior can vary between attempts, as Anthropic’s evaluation guide explains. For each case, report successes out of all valid trials, including runs where the agent stalled, timed out or failed, and list failures by category. Report invalid test-environment runs separately. A case that succeeds once and fails on the next attempt is not consistently handled. Choose the number of repetitions around the consequences of failure and the confidence needed for the decision; there is no universal count that proves readiness.
Compare the proposed release with the current version on the same set. Review authority breaches, missing or duplicated work, unsupported statements and useful handovers separately. Also examine completion time and operator corrections. A stronger overall average can hide a regression in the very case that should block release.
Turn the evidence into a release decision
Agree the decision rules before running the candidate. For this hypothetical workflow, any observed attempt to access unauthorized data, even if blocked, or to send a customer message blocks release of that behavior. Missing and duplicated tasks require correction and retesting. The business owner sets acceptable completion and handover criteria for ordinary requests, and the sales operations lead confirms that the review queue can be staffed.
For example, suppose a candidate handles clear requests and ambiguity correctly but duplicates work after a timeout. Hold the create-task release while the recovery path is fixed. A draft-only mode may be a narrower option if it has its own passing tests and an operator explicitly owns saving the task. Changing the permitted action changes the release being approved; it is not a way to relabel a failed test as a pass.
The decision record should identify the version, case coverage, repeated-run results, unresolved failures, permitted scope, reviewer and operating owner. Passing the set supports a bounded decision, not a guarantee about unseen requests. After a limited launch, turn actual corrections and incidents into new cases and rerun the relevant set when the model, tools or business rules change.
Sources and further reading
Define the evidence your agent needs to pass.
Bring one workflow, its connected systems and a few difficult requests. We can discuss the permitted actions, test cases and acceptance criteria for a focused release.
Discuss an AI agent workflow
