A worked example, not a customer claim. The scenario, workflow and deliverables below are illustrative. No completed engagement or measured outcome is represented.
01 / Agent evaluation
The situation: a convincing answer can hide a failed action.
Consider a support agent that can retrieve account information and prepare a service change. A response may sound successful even if the wrong tool was selected, the arguments were incomplete or the state change never occurred. Evaluating only the final message would leave those failures invisible.
This illustrative study describes a proposed test workflow, not a real customer result. Test actions should run in a controlled environment with synthetic or appropriately authorized data and safeguards against unintended production changes.
02 / Agent evaluation
Specify the starting state and allowed actions.
Each scenario defines the user request, available evidence, initial state, tool contract and permitted outcome. Include tasks that require clarification, situations with incomplete retrieval and recoverable tool errors. An agent should not receive credit for inventing missing information to finish quickly.
Record what constitutes success and what would be a critical failure. For an action-taking task, inspect the resulting state rather than relying on a self-reported success message. Keep approval requirements explicit: a technically valid action may still be inappropriate without the required user authorization.
- Initial state
- Known records, permissions, tool availability and relevant task context.
- Expected behavior
- Allowed actions, required clarification and acceptable recovery paths.
- Final state
- Observable evidence of completion, safe refusal or a justified request for help.
03 / Agent evaluation
Review the trajectory in separate dimensions.
Capture the observable interaction trace: retrieved material, tool selections, arguments, results and state changes. This is a record of external behavior, not a requirement to expose private model reasoning. Give reviewers sufficient evidence to distinguish a planning error from a tool or data failure.
Use a rubric that separates task completion from process defects. Several valid paths may exist, so avoid rewarding one exact sequence unless the workflow truly requires it. Have a specialist adjudicate disputed or consequential cases and preserve the reason for the decision.
| Dimension | What to inspect |
|---|---|
| Retrieval | Whether the evidence is relevant, sufficient and used appropriately. |
| Tool use | Tool selection, argument validity and authorization boundaries. |
| Recovery | Handling of errors, unavailable tools and missing information. |
| State | The actual result of attempted actions and consistency with the request. |
| Final response | Whether the response accurately represents the observed outcome. |
04 / Agent evaluation
Turn failures into an actionable backlog.
The proposed handoff joins scenario-level outcomes with issue categories, trace references and recommended owners. Separate problems in prompts, retrieval content, tool contracts and application state handling. Keep unresolved cases visible instead of forcing every trace into pass or fail.
Rerun a stable regression set after a change and introduce new cases for newly discovered failure modes. Record the test configuration so comparisons remain meaningful. No performance improvement is claimed here; the benefit of the workflow is a clearer basis for deciding what to fix and retest.