An agent can return a plausible but wrong answer while every server reports a successful request. Amazon’s new observability view puts that quality failure beside the trace that produced it.
Amazon CloudWatch Omni became generally available on September 23, 2026. It brings application telemetry and AI-agent traces into a dedicated web experience and an IDE extension. AWS lists US East (N. Virginia), US West (Oregon), and Europe (Ireland) as launch regions. A local extension for VS Code, Cursor, and Kiro is free to use without an AWS account; cloud telemetry and evaluation have their own service costs.
A failure that an error chart misses
Imagine a support agent that retrieves the right ticket but uses an old policy excerpt. The API call succeeds, response time stays normal, and the answer is grammatically sound. A conventional uptime dashboard sees no outage. The useful record is the trace: user request, retrieval step, model call, tool result, and final answer. An evaluation can then ask whether the answer is grounded in the approved policy.
CloudWatch Omni displays operational signals such as errors, latency, and token usage alongside evaluation scores. AWS documents trace, session, and tool-call levels for scoring, including built-in and custom evaluators. Teams still need to decide what correctness means for their product and test whether an automatic judge agrees with human reviewers on difficult cases.
Where the telemetry goes
Omni reads data ingested into CloudWatch. Existing CloudWatch instrumentation can appear there; other workloads can send OpenTelemetry data through an OTLP endpoint. AWS documents application paths for EC2, ECS, EKS, Azure VMs and Azure AKS, and agent paths for AgentCore and several AWS runtimes. A team should verify its collector, permissions and endpoint for the environment it actually runs.
AWS says Omni can show telemetry across accounts and Regions when CloudWatch centralization rules bring that data together. The Omni interface doesn’t aggregate it independently. That distinction matters when a pilot looks empty: the problem may be collection or centralization, not the dashboard.
Trace, score, correct, repeat.
- Instrument one agent so a run contains the request, model, and tool spans needed for diagnosis.
- Choose a known failure and find its trace in Omni.
- Turn representative traces into a fixed evaluation dataset.
- Score the current agent, then change one prompt, model, or tool.
- Compare scores and human review before promoting the change.
- Monitor a sample of live traffic for regression.
For multi-turn conversations, AWS says Omni groups turns using the OpenTelemetry session.id span attribute. Custom agents that omit it can make one conversation appear as unrelated traces. Add a stable session ID to the root span of each turn before relying on session-level analysis.
Natural-language investigation still needs evidence.
The Omni agent can turn a plain-language question into a SQL or PromQL query and help investigate telemetry. That can shorten the path to a useful graph. An incident report should preserve the actual query, time range, data source, and trace links so another engineer can reproduce the conclusion. AWS’s root-cause and productivity benefits are company claims until a team measures them in its own environment.
Budget for collection and evaluation
The free local IDE extension is not a blanket free tier for hosted observability. Price a pilot from trace and log ingestion, retention, queries, and evaluator runs. Decide which agent payloads to record before enabling full traces; prompts and tool results can contain sensitive information. A sampling plan and redaction rule should be part of the instrumentation work, not an afterthought.
Our GitHub Copilot OpenTelemetry guide explains the same trace vocabulary in a coding agent. The Taste-Bench analysis shows why a successful run needs a quality measure beyond completion status.
Primary sources
- AWS: CloudWatch Omni GA announcement
- Amazon CloudWatch Omni documentation
- AWS: Monitor AI agents in Omni
- CloudWatch Omni pricing
Checked September 24, 2026. Service capabilities and availability are from AWS; the support-agent example is illustrative.
