Evaluation becomes actionable when every scenario connects to a verifiable final state, a policy and an explicit production decision.
An AI agent team should not enter production because it earns a good average score on a test set. An average can offset one critical failure with several easy successes: the system handles nine requests correctly, then updates the wrong record on the tenth. A release decision must combine business outcome success, trajectory quality, policy compliance and non-negotiable gates for sensitive effects.
Useful evaluation therefore does not compress the entire system into one number. It builds an evidence case for the scope that will actually be deployed. Every scenario describes an initial state, a request, available tools, a verifiable outcome and prohibited behaviors. The go/no-go gate then applies thresholds by risk class, with an immediate rejection when a critical rule fails.
An average hides the failures that matter
Agents are harder to evaluate than single-turn answers. They select tools, modify an environment, transfer tasks and react to intermediate results. Two runs of the same scenario may follow different paths and return similar responses even though one respected the mandate while the other attempted a prohibited action.
An aggregate score then mixes separate questions. Was the final outcome correct? Did the agent use the right identity? Did it consult an authorized source? Did it escalate at the right point? Does the real system state match what the response claims? These answers belong to different evidence families and should not automatically compensate for one another.
The Atlensia recommendation is to reserve averages for gradual qualities such as clarity or synthesis quality. Security, authority, confidentiality and business-integrity rules become binary gates. A system cannot offset a data disclosure with more polished writing on unrelated cases.
An evaluation scenario describes complete work
Anthropic defined several useful objects for agent evaluation in January 2026. A task contains an input and success criteria; a trial is one attempt at that task; a grader applies checks; a trace records the trajectory; and an outcome is the final state of the environment. This distinction prevents teams from evaluating only what the agent says it accomplished.
For an autonomous team, the scenario must also state the mandate. It identifies each agent's role, accessible tools, initial data, simulated human approvals, time limits and escalation conditions. The test therefore covers the model, orchestration and environment together. Changing a connector, handoff rule or prompt may alter behavior even when the model stays the same.
A sound test case is decidable by two domain experts. If they cannot agree on the expected result, the first problem is the specification. Ambiguity creates measurement noise and encourages the team to tune the grader until it produces the preferred score. A reference solution or known final state verifies that the scenario is actually solvable.
The initial set should contain common cases, boundaries and refusals. Testing only cases where the agent should use a tool creates a system that learns to use tools too often. Testing only complete cases ignores whether the team asks for missing evidence or returns ownership. Evaluation must cover the expected behavior and its opposite when conditions are not met.
Four evidence families cover the system
Deterministic checks answer questions with objective answers. Contextual judgments assess meaning, usefulness or domain expertise. Some tests isolate a component while others execute the whole process. The following matrix shows why no single evaluation technique is sufficient.
Atlensia diagram: the production decision combines four evidence families, then applies critical prohibitions independently of the average score.
The component and deterministic quadrant covers schemas, types, tool contracts, state transitions and permissions. It can confirm that a function rejects an unknown field or that a handoff preserves the same task identifier. These tests are fast and precise, but they do not prove that the team assembles the components correctly when facing a real request.
The end-to-end and deterministic quadrant checks business state after execution. Did an invoice remain pending instead of being paid? Does a ticket have the right status? Was the requested record, and only that record, updated? This quadrant is often the most important because it compares the agent's claim with reality in the system of record.
Contextual judgment completes the evidence. At the component level, it can assess routing relevance or the clarity of a clarification question. End to end, it examines outcome usefulness, evidence quality and whether escalation was appropriate. A model grader can accelerate this work, but its rubric must stay explicit and be calibrated against expert human decisions.
The business outcome outranks the final response
An agent can announce that a reservation was created when no database record exists. It may also return a cautious final answer after attempting a prohibited call. Grading only the final message rewards a convincing narrative instead of the work that actually happened.
The outcome grader should query the system of record or a controlled replica. It checks the created object, changed fields, absence of additional effects and consistency with the request. For read-only work, it may confirm that cited references exist and support the answer. For write tasks, it compares state before and after, including objects that were not supposed to change.
The final response still matters for a different reason. It should honestly describe the state achieved, identify uncertainty and explain what still requires a human decision. The test then connects communication with reality: a silent failure is penalized, as is an unsupported claim of success.
This distinction extends the method for measuring the value of an autonomous team. Preproduction evaluation determines whether the system follows a behavioral contract in controlled scenarios. Operational measurement later determines whether that behavior creates durable value under real conditions. Neither replaces the other.
The trajectory explains fragile successes
Final state does not tell the whole story. An agent may reach the correct result after unnecessary calls, accessing excessive data or bypassing an approval step without a visible consequence in the test. The trace makes those accidental successes visible before they become incidents.
The OpenAI Agents SDK documents a trace as a complete operation composed of spans for generations, tool calls, handoffs, guardrails and custom events. That instrumentation provides evaluation material, but it is not a verdict by itself. The team must turn selected events into assertions: only an allowed tool was called, the expected identity was used, approval occurred before action and no secret appeared in an unapproved destination.
The opposite mistake is requiring one exact sequence of clicks or calls when several paths are valid. Anthropic recommends emphasizing what the agent produced instead of overfitting tests to a rigid trajectory, because an agent may find a correct method the designer did not anticipate. The trace should become prescriptive only for invariants that protect the process, not every execution detail.
A sound trajectory assertion therefore states a business constraint. “Approval must exist before sending” is robust. “The agent must call A, then B, then C” is robust only if that exact order is truly required. This distinction preserves adaptability without sacrificing guarantees.
Repeated trials expose variability
One run is not enough for a probabilistic system. A scenario may pass once and fail when wording, result order or generation changes slightly. Repeated trials reveal that variability and prevent a lucky success from deciding the launch.
There is no universal trial count. It depends on scenario frequency, failure severity, environment cost and the confidence required. A reversible, low-risk action can tolerate a less precise estimate. A financial action or data exposure requires more repetition and safety gates that do not depend on an average statistical result.
Reports should preserve the distribution. Overall pass rate is supplemented with the frequency of each failure class, variation across trials, time, cost and tool-call count. An average improvement may conceal a new rare but severe failure. The team therefore inspects individual cases and traces before declaring one version better.
This practice also makes model comparison more honest. Model A may be more consistent on rules but weaker on ambiguous cases; model B may produce stronger synthesis with higher variance. The right choice depends on the role assigned within the autonomous team, not a universal leaderboard.
The release gate applies risk-specific rules
A go/no-go decision becomes defensible when every failure family has an owner, measure and consequence. The thresholds below are decision forms, not universal numerical values. Each organization calibrates them to its scope and risk tolerance.
| Evaluation family | Expected evidence | Release rule | Response to failure |
|---|---|---|---|
| Authorization | Identity, scope and downstream decision | No action without a valid right | Immediate no-go and control fix |
| Confidentiality | Destinations and data actually transmitted | No prohibited disclosure | No-go and data-path analysis |
| Business integrity | Before/after diff in the system of record | No out-of-scope object changed | No-go and narrower capability |
| Human approval | Timestamp and version of approved proposal | Approval required before sensitive effect | No-go and sequence correction |
| Task outcome | Final state and scenario-specific criteria | Threshold per scenario, not only globally | Fix or narrow the scope |
| Exception handling | Escalation state, owner and reason | Every dead end returns to an owner | Fix the handoff or recovery queue |
| Content quality | Calibrated rubric and references | Level appropriate to intended use | Improve context, model or review |
| Cost and latency | Measures per trial and tail cases | Budget compatible with service expectations | Optimize, route or limit turns |
| Robustness | Variants, refusals and adversarial trials | Stable behavior within expected range | Expand cases and add controls |
The first four rows are typically critical gates for a connected agent. One failure invalidates release for that scope even when other scores are high. Gradual dimensions may use a threshold, comparison with a reference version or human review, provided the decision rule was defined before results were observed.
A no-go does not always mean abandoning the product. It may narrow the scope, remove a capability, add human validation or route selected cases into a deterministic workflow. The gate chooses the deployment form supported by the evidence instead of merely producing an abstract green or red light.
The test environment must resemble production
A credible suite runs the agent with tools, rules and states close to the intended scope. An oversimplified simulation misses pagination, permissions, real formats and latency failures. Testing directly in production, however, exposes data and may create unintended effects. The practical compromise is a controlled environment with faithful doubles and separately verified integration points.
Every trial starts from a clean state. Files, caches, conversations and records produced by one attempt must not help the next. Anthropic notes that shared state can correlate failures or artificially raise performance. Isolation is therefore part of measurement quality, just like the graders.
Deterministic orchestration tests can replace the model with scripted responses to verify handoffs, retries, guardrails and tool calls. The OpenAI Agents SDK testing documentation distinguishes these application-owned behaviors from provider behavior and integrations that require a real adapter or controlled network environment. Testing layers separately speeds diagnosis because an orchestration defect is not confused with model variation.
The dataset is versioned with the prompt, model configuration, tools and policies. Without that snapshot, two results are not comparable. An apparent improvement may come from changing the cases or grader rather than the system being evaluated.
Model graders require domain calibration
Model-based rubrics help when several answers may be valid. They can assess relevance, completeness, caution or reasoning quality faster than exhaustive manual review. They must not become an opaque source of truth.
The rubric states observable criteria and includes positive and negative examples. Domain experts grade a sample, then the team compares their decisions with the model judge. Disagreements expose a vague criterion, position bias, stylistic preference or inability to verify evidence. The judge is recalibrated and checked regularly, especially after a model change.
Code and systems of record remain preferable for critical rules. A model judge should not decide whether a payment exists, a field changed or an identity held a permission. It evaluates genuinely subjective dimensions and leaves an explanation a reviewer can inspect.
Human review retains a separate role. Reading a sample of traces confirms that failures are fair and successes do not hide a workaround. That review generates new cases instead of mechanically approving a dashboard.
Release begins a new evaluation phase
Passing the release gate does not prove the team will work in every situation. It establishes that a documented scope has enough evidence for controlled deployment. The launch record states allowed cases, limits, monitored signals, rollback mechanism and decision owner.
Preproduction scenarios then become a regression suite. Each incident, unexpected escalation and user report can produce a new case after sensitive data is removed or protected. NIST recommends testing, evaluation, verification and validation before deployment and regularly during operation, with measures connected to mapped risks.
Production monitoring completes offline evaluation. It detects distribution shifts, new integrations and behavior the test set did not anticipate. A version may retain its historical suite score while becoming less suitable for current cases. The decision to remain in production should therefore be revisited when context, models, tools or policies change.
Conclusion
Evaluating an AI agent team means proving behavior in context, not searching for a flattering score. The decision case combines component tests, business outcomes, traces, calibrated judgments and repeated trials. Gradual qualities may be aggregated; critical violations remain independent gates.
The practical next step is to choose one business scope and write its scenarios before tuning the agent. Every scenario names the initial state, mandate, expected outcome, prohibitions and verification method. This foundation turns “the agent feels ready” into a go/no-go decision that product, domain, security and operations teams can actually examine.
Primary sources and references
Anthropic, Demystifying evals for AI agents, January 9, 2026
OpenAI API, Evaluation best practices, active documentation in September 2026
OpenAI API, Working with evals, active documentation in September 2026
OpenAI Agents SDK, Tracing, active documentation in September 2026
OpenAI Agents SDK, Testing, active documentation in September 2026
NIST, AI Risk Management Framework Core, version 1.0, 2023
Atlensia, platform for autonomous enterprise teams, 2026