Managing runtime evaluations (Preview)

Validate deployed agents against datasets on demand or on a recurring schedule.

An agent that passed its tests at design time can still fail in production. Models get updated, inputs drift, and the tools your agent calls change without warning. If you only evaluate before you ship, your users are the ones who find these failures.

Runtime evaluations run against the deployed agent itself, so the results reflect what's in production, not a design-time copy. Run them after every change and on a schedule to catch regressions before your users do.

To evaluate an agent while you're still building it, see Evaluations.

Evaluate what the agent did, not only the output

An agent can return the right answer by the wrong route: it calls a tool it shouldn't have, skips a step it was supposed to take, or loops three times before it gets there. A check on the final output won't catch any of that, and in production those are the failures that cost you.

Each data point is scored by the evaluators linked to its dataset. You add evaluators in the Evaluators view. They come in three categories, and each one catches a different kind of failure, so combine more than one.

Output validation evaluators run deterministic checks. They're cheap and give the same result every time.

• Contains Text: Passes when the output contains the text you specify.
• Exact Match: Passes when the output exactly matches the expected output.
• JSON Similarity: Passes when the JSON structure of the output is similar to the expected one.

LLM judge evaluators use an LLM to score the run, so they accept answers that are correct but worded differently.

• LLM Judge Output: Judges whether the output matches the expected output in meaning. The default LLM Judge Evaluator uses this type.
• LLM Judge Strict JSON: Judges whether the output strictly matches the expected JSON structure.
• LLM Judge Trajectory: Judges whether the run followed the expected behavior. You can't use it in scheduled runs yet.
• LLM Judge Simulation: Judges the execution path of a run that uses simulated components.

Tool call evaluators check how the agent used its tools.

• Tool Call Arguments: Checks that the agent called tools with the correct arguments.
• Tool Call Count: Checks how many times the agent called each tool.
• Tool Call Order: Checks that the agent called tools in the expected sequence.
• Tool Call Output: Checks that the tool outputs match the expected values.

Every evaluator result comes with a score and a full execution trace, so you can see where a run went wrong. LLM judge evaluators also explain their score with a justification. Treat the results as a to-do list for improving your agent: export your runs and use CLI skills to analyze them. For example, you can:

• Rank the failures that matter most, so you fix the highest-impact problems first.
• Check whether your datasets cover every instruction in the agent's prompt, and add data points for the ones they miss.
• Find use cases your agent handles in production that your datasets don't test yet.

Build datasets from real traffic

A dataset is a set of data points. Each one holds the inputs you send to the agent and the output you expect back. You can write data points by hand, but you don't have to. There are three other sources, and production runs are the most representative because they include inputs you didn't think to test:

• Production runs. Completed runs of your deployed agent can go directly into a dataset. Failures and odd edge cases are worth adding first, because those are the cases to guard against. See Add a production run to a dataset.
• Generated examples. Use CLI skills to generate data points when you need coverage the agent hasn't seen yet. See Agents in the Coding Agents guide.
• Your business teams. Use CLI skills to import test cases (JSON or CSV) from the people who know what a correct answer looks like. For judgment-heavy tasks, they should define the expected output, trajectory, or escalation paths.

Before you rely on a data point, review its expected output. The data point form matches your agent's input and expected output schemas, and it warns you when the expected output doesn't match the keys in the agent's output.

Simulate tools with side effects

An evaluation run sends each data point to your deployed agent. If the agent's tools are live, the run triggers them: it sends the email, writes the record, and creates the escalation for a human to handle. If you don't want those downstream effects in production, simulate every tool and escalation that changes something outside the agent. This includes human-in-the-loop (HITL) steps, so an evaluation doesn't put work in a person's queue.

Each simulated component starts with its output schema filled in. You can simulate it in two ways:

• Static mock. The component returns the same response on every run, for example a HITL step that always approves, or a tool that always returns the same output. Mocks are precise and cheap, so use them whenever a fixed response is enough.
• Generated response. You describe the response you want the agent to see, and a new one is generated each time the evaluation runs. Use this when the agent needs to handle varied responses.

Run on every change, and on a schedule

You can start an evaluation run on demand or from a schedule.

• On demand: Run your datasets after every change to something the agent depends on, such as the model, the prompt, a tool, or a connected system, to catch a regression before it spreads. Use the same datasets at design time and at runtime, and run them at design time first.
• On a schedule: A recurring run catches drift. The right frequency depends on how fast your inputs and dependencies change. A high-volume agent that touches changing systems needs a tighter cadence than one that doesn't.

Schedule an evaluation

A schedule runs a whole dataset. To create one:

1. Under Agent Instance Management, select Agents.
2. Select the deployed agent.
3. Select the Evaluations tab.
4. In the Schedules view, select New schedule.
5. Set the start date and the end date. Times are shown in your local time zone.
6. Set the recurrence, that is, how often the dataset runs.
7. Choose the dataset to run. The agent's first dataset is selected by default.
8. 保存计划。

Each scheduled run uses the dataset as it is when the run starts. If you add, edit, or remove data points, the next run picks up the changes automatically, so you don't need to reschedule.

After you create a schedule, you can pause and resume it, change how often it runs, or delete it. The Schedules view lists only the schedules in the deployed agent's folder.

Read the results

Open a run in the Eval runs view to see:

• The run's status and overall score
• How many data points passed
• The status and score of each data point

An overall score tells you whether something changed. It doesn't tell you what. Selecting a data point shows:

• Justification: Why the evaluator gave that score.
• Input, Expected output, and Expected vs actual output: The data point's values, next to what the agent actually produced.
• Trace: What the agent did during the run. The trace starts at the Evaluation set run span and includes the agent run, the agent output, and an Evaluator span for each evaluator.

When a data point's score drops, either the agent is wrong or the expected output is. If the agent is wrong, that's a regression. If the expected output is wrong, the data point needs fixing so the dataset stays trustworthy.

Where runtime evaluations live

Open Agent Registry > Agents and select a deployed agent. Its Evaluations tab has four views:

视图	用途
数据集	Create and manage data points, and see each dataset's linked evaluators and last update.
评估者	Choose how each data point is scored. If your agent has none, default evaluators are created, and the default dataset is linked only to the LLM Judge Evaluator.
Eval runs	Review past runs, including scores and traces.
计划	Run datasets on a recurring basis.

What you can do on the Evaluations tab depends on your permissions in the deployed agent's folder. Without them, editing and run actions are hidden or unavailable, and the tab is read-only.

Runtime evaluations are stored separately from the agent package. You can change datasets, data points, and evaluators without republishing the agent. Files attached to data points live in Orchestrator storage buckets, and you manage them from the Evaluations tab, not the Orchestrator Buckets page.

Add a production run to a dataset

You can turn a completed run of your deployed agent into a data point. The run's output becomes the data point's expected output, so check that it's correct before you add it. A wrong answer saved here becomes the answer future runs are scored against.

开始之前

• Create at least one dataset in the Datasets view of the Evaluations tab.
• Make sure the agent has at least one completed run.

步骤

1. Under Agent Instance Management, select Agents.
2. Select the deployed agent.
3. Select the Review runs tab.
4. In the run's row, select the icon in the Add to dataset column. If the run's feedback dialog is open, select Add to dataset there instead.
5. In Dataset, choose a dataset. The first one is selected by default.
6. Optionally, enter a Data point name. If you leave it blank, the data point is named after the trace, for example Trace 083869d6.
7. Review the Input and Expected output values.
8. Select Add data point.

A confirmation message names the dataset, and the data point appears in it on the Evaluations tab.

限制

• Scheduled runs don't support the LLM Judge Trajectory evaluator yet.
• Online evaluations aren't available.
• Maestro runtime evaluations aren't available.