AWS has introduced the Agent Evaluation Metric, a turn-level method for finding the first wrong response or tool call in a multi-step conversation. Its most useful idea is simple: downstream failures should not all be counted as new root causes.
The AWS technical post published September 10, 2026 describes the first AEM dimension, correctness. It divides correctness into truthfulness and completeness, applies them to both responses and actions, then attributes later failures to the earlier turn that corrupted the chain.
Start with a five-turn failure
Imagine that an agent is building a sales report. The correct tool call should request revenue for the EU, but turn two requests profit. Turns three through five faithfully operate on the wrong data. An outcome-only evaluator marks the whole task as failed. AEM marks turn two as the root cause and the later turns as inherited failures.
That distinction changes the engineering queue. Without attribution, four failed turns can become four tickets. With attribution, the team fixes the parameter-selection problem at turn two and reruns the entire trajectory.
AWS Agent Evaluation Metric uses the same questions for words and tools
| Turn type | Truthfulness asks | Completeness asks |
|---|---|---|
| Agent response | Is the statement consistent with the reference? | Did the response address every required part? |
| Tool action | Are the supplied parameter values semantically correct? | Are all required parameter keys present, with no unexpected extras? |
A structural layer checks whether the agent selected the correct tool and operation before the parameter checks run. This produces a failure taxonomy that includes tool_mismatch, action_mismatch, missing_parameters, extra_parameters, inconsistent_parameter_values, inconsistent_response and incomplete_response.
The label that makes the metric operational
A failure receives prior_action_failed when it is wrong only because it consumed output from an earlier failed turn. AWS treats those instances as cascading effects. The first failure without that label becomes the likely root cause.
The example result reports a success rate of 0.2, first failure at turn two, one root cause and three cascading failures. AWS explicitly describes the example as illustrative. It is not a production performance result for Amazon Quick or another deployed system.
AEM requires a gold conversation, not just a final answer
The evaluation dataset must annotate the expected response and expected tool call for every relevant turn. This is more expensive than recording whether the final task passed, but it creates diagnostic evidence. Teams can bootstrap labels with a stronger model, but AWS recommends human review because the reference defines what correct means.
Semantic comparison allows equivalent phrases such as New York City and NYC to pass. AWS shows 0.5 as a starting similarity threshold and says it is not tuned. A customer-support assistant, financial agent and medical workflow should not inherit the same threshold without domain testing.
Three decisions the default score should not make for you
- Turn weighting: an incorrect payment amount should count more than a harmless wording omission.
- Gating: one critical safety or authorization failure may need to fail the whole trajectory.
- Judge calibration: embedding similarity and an LLM judge have different cost, opacity and error patterns.
A plain average of passing turns can hide those differences. Keep the decomposed fields even if leadership wants one headline number.
A minimum regression suite for a production agent
- Collect successful, failed and ambiguous trajectories from the real workflow.
- Annotate the expected response, tool, action and required parameters at each critical turn.
- Measure agreement between the automated evaluator and human reviewers.
- Set different gates for ordinary and high-impact actions.
- Track the first-failure category by model, prompt, tool version and chain length.
- Rerun the suite after any model, tool schema or retrieval change.
The metric complements execution evidence
AEM evaluates whether the trajectory matches a reference. It does not prove that the executed code was safe or that the external system returned truthful data. Our Copilot code-review firewall analysis covers the separate controls needed when an agent runs builds and tests. Our ToolGrad guide explains another part of the lifecycle: creating tool-use training examples before evaluation begins.
The practical verdict
The AWS Agent Evaluation Metric is valuable because it preserves the path to a failure. Teams should adopt the decomposition and attribution pattern, then calibrate the judge, thresholds and business-critical gates with their own labelled trajectories.
Primary source
Checked September 14, 2026. The framework and examples are from AWS. The post does not publish independent production benchmarks for AEM.
