# Managing runtime evaluations (Preview)

> Runtime evaluations for validating deployed agents against datasets of inputs and expected outputs, on demand or on a recurring schedule.

An agent that passed its tests at design time can still fail in production. Models get updated, inputs drift, and the tools your agent calls change underneath it. If you only evaluate before you ship, you find out about these failures from your users.

Runtime evaluations run against the deployed agent itself, so the results describe what is in production, not a design-time copy. Running them after every change and on a schedule surfaces regressions early, the way a failing test does in CI.

To evaluate an agent while you're still building it, see [Evaluations](agent-evaluations.md).

## Evaluate what the agent did, not only the output

An agent can return the right answer by the wrong route: it calls a tool it shouldn't have, skips a step it was supposed to take, or loops three times before it gets there. A check on the final output won't catch any of that, and in production those are the failures that cost you.

Each data point is scored by the evaluators linked to its dataset. Use more than one kind:

* **Output** The **LLM Judge Evaluator** scores how closely the agent's output matches the expected output in meaning. It's the default, and it's a reasonable first check. It's also the weakest on its own, because a fluent wrong answer can still score well.
* **Tool calls** Check that the agent called the right tools with the right arguments. <!-- confirm: name and behavior of the tool call evaluator -->
<!-- * **Trajectory.** Check the sequence of steps the agent took, not just where it ended up. confirm: name and behavior of the trajectory evaluator -->

Every evaluator result comes with a justification and a full execution trace, which show why a score dropped.

## Build datasets from real traffic

A dataset is a set of data points. Each one holds the inputs you send to the agent and the output you expect back. You can write data points by hand, but you don't have to. There are three other sources, and production runs are the most representative because they include inputs you didn't think to test:

* **Production runs.** Completed runs of your deployed agent can go directly into a dataset. Failures and odd edge cases are worth adding first, because those are the cases to guard against. See [Add a production run to a dataset](#add-a-production-run-to-a-dataset).
* **Generated examples.** Use CLI skills to generate data points when you need coverage the agent hasn't seen yet. See [Agents](https://docs.uipath.com/coding-agents/standalone/latest/user-guide/build-agents) in the Coding Agents guide.
* **Your business teams.** Use CLI skills to import test cases (JSON or CSV) from the people who know what a correct answer looks like. For judgment-heavy tasks, they should define the expected output, trajectory, or escalation paths.

Before you rely on a data point, review its expected output. The data point form matches your agent's input and expected output schemas, and it warns you when the expected output doesn't match the keys in the agent's output.

![Adding a completed run from the Review runs tab of a deployed agent to a dataset](https://dev-assets.cms.uipath.com/assets/images/agents/agents-runtime-evals-build-datasets-d7fb85fd.gif)

## Simulate tools with side effects

An evaluation run sends each data point to your deployed agent. If the agent's tools are live, the run triggers them: it sends the email, writes the record, and creates the escalation for a human to handle. If you don't want those downstream effects in production, simulate every tool and escalation that changes something outside the agent. This includes human-in-the-loop (HITL) steps, so an evaluation doesn't put work in a person's queue.

Each simulated component starts with its output schema filled in. You can simulate it in two ways:

* **Static mock.** The component returns the same response on every run, for example a HITL step that always approves, or a tool that always returns the same output. Mocks are precise and cheap, so use them whenever a fixed response is enough.
* **Generated response.** You describe the response you want the agent to see, and a new one is generated each time the evaluation runs. Use this when the agent needs to handle varied responses.

![Adding a data point in the Datasets view and simulating the agent's tools and escalations](https://dev-assets.cms.uipath.com/assets/images/agents/agents-runtime-evals-simulate-tools-f6192f12.gif)

## Run on every change, and on a schedule

You can start an evaluation run on demand or from a schedule.

* **On demand:** A run after each change to something the agent depends on, such as the model, the prompt, a tool, or a connected system, finds a regression before it spreads. Use the same datasets at design time and at runtime, and run them at design time first.
* **On a schedule:** A recurring run catches drift. The right frequency depends on how fast your inputs and dependencies change. A high-volume agent that touches changing systems needs a tighter cadence than one that doesn't.

### Schedule an evaluation

A schedule runs a whole dataset. To create one:

1. Under **Agent Instance Management**, select **Agents**.
2. Select the deployed agent.
3. Select the **Evaluations** tab.
4. In the **Schedules** view, select **New schedule**.
5. Set the start date and the end date. Times are shown in your local time zone.
6. Set the recurrence, that is, how often the dataset runs.
7. Choose the dataset to run. The agent's first dataset is selected by default.
8. Save the schedule.

Each scheduled run uses the dataset as it is when the run starts. If you add, edit, or remove data points, the next run picks up the changes automatically, so you don't need to reschedule.

After you create a schedule, you can pause and resume it, change how often it runs, or delete it. The **Schedules** view lists only the schedules in the deployed agent's folder.

![Creating a schedule from the Schedules view, with its start and end dates, recurrence, and dataset](https://dev-assets.cms.uipath.com/assets/images/agents/agents-runtime-evals-schedule-ae406707.gif)

## Read the results

Open a run in the **Eval runs** view to see:

* The run's status and overall score
* How many data points passed
* The status and score of each data point

An overall score tells you whether something changed. It doesn't tell you what. Selecting a data point shows:

* **Justification:** Why the evaluator gave that score.
* **Input**, **Expected output**, and **Expected vs actual output:** The data point's values, next to what the agent actually produced.
* **Trace:** What the agent did during the run. The trace starts at the **Evaluation set run** span and includes the agent run, the agent output, and an **Evaluator** span for each evaluator.

When a data point's score drops, either the agent is wrong or the expected output is. If the agent is wrong, that's a regression. If the expected output is wrong, the data point needs fixing so the dataset stays trustworthy.

## Where runtime evaluations live

Open **Agent Registry** > **Agents** and select a deployed agent. Its **Evaluations** tab has four views:

| View | What it's for |
|---|---|
| **Datasets** | Create and manage data points, and see each dataset's linked evaluators and last update. |
| **Evaluators** | Choose how each data point is scored. If your agent has none, default evaluators are created, and the default dataset is linked only to the **LLM Judge Evaluator**. |
| **Eval runs** | Review past runs, including scores and traces. |
| **Schedules** | Run datasets on a recurring basis. |

![The Evaluations tab of a deployed agent, open on the Schedules view, with the Datasets, Evaluators, Eval runs, and Schedules views at the top](https://dev-assets.cms.uipath.com/assets/images/agents/agents-runtime-evals-where-they-live-bd911181.png)

What you can do on the **Evaluations** tab depends on your permissions in the deployed agent's folder. Without them, editing and run actions are hidden or unavailable, and the tab is read-only.

Runtime evaluations are stored separately from the agent package. You can change datasets, data points, and evaluators without republishing the agent. Files attached to data points live in Orchestrator storage buckets, and you manage them from the **Evaluations** tab, not the Orchestrator **Buckets** page.

## Add a production run to a dataset

You can turn a completed run of your deployed agent into a data point. The run's output becomes the data point's expected output, so check that it's correct before you add it. A wrong answer saved here becomes the answer future runs are scored against.

### Before you start

* Create at least one dataset in the **Datasets** view of the **Evaluations** tab.
* Make sure the agent has at least one completed run.

### Steps

1. Under **Agent Instance Management**, select **Agents**.
2. Select the deployed agent.
3. Select the **Review runs** tab.
4. In the run's row, select the icon in the **Add to dataset** column. If the run's feedback dialog is open, select **Add to dataset** there instead.
5. In **Dataset**, choose a dataset. The first one is selected by default.
6. Optionally, enter a **Data point name**. If you leave it blank, the data point is named after the trace, for example `Trace 083869d6`.
7. Review the **Input** and **Expected output** values.
8. Select **Add data point**.

A confirmation message names the dataset, and the data point appears in it on the **Evaluations** tab.

## Limitations

* Trajectory LLM-judge evaluators aren't available for deployed agents yet.
* Online evaluations aren't available yet.
* Maestro runtime evaluations aren't available yet.