# Evaluations

> Set up evaluations for a workflow: datasets, evaluation targets, and evaluators, then run them and use the results during development.

Evaluations are automated quality checks on the output of a workflow. Expected outcomes are grouped into a dataset, and each run is scored by the evaluators attached to it. The **Datasets**, **Evaluators**, and **Eval runs** tabs of the execution panel hold the three parts.

## Concepts

- Dataset: a named collection of data points that share an evaluation target and a set of evaluators. A dataset lists its evaluation target, its evaluators, and its number of items.
- Data point: a single case, made of input values and the expected output for those inputs. A data point can be written by hand or captured from an actual run, as [Capturing a data point from a run](#capturing-a-data-point-from-a-run) describes.
- Evaluator: a check that compares the actual output with the expected output and produces a score. A boolean evaluator reports pass or fail.
- Evaluation run: one run of a dataset, which scores every data point with the selected evaluators.
- Score: the result an evaluator reports for a data point.

## Evaluation targets

A dataset has one evaluation target, which is either the whole workflow or a single node:

- **Whole process**: tests the path from start to end. It is the recommended target. The dataset names an entry point, a trigger such as **Manual trigger**, and each data point supplies the trigger's inputs and the workflow's expected output.
- **Agent node**: tests one agent node in isolation.
- **Document processing**: evaluates a document processing node.

The node targets are unavailable in a workflow that has no agent nodes or document processing nodes. A single dataset cannot mix targets.

## Evaluators

The **New evaluator** button on the **Evaluators** tab lists the evaluator types in three groups. Each type is tagged **Flow**, for a whole-workflow target, or **Agent**, for an agent node target, or both.

### Output validation

These evaluators compare the output with the expected output exactly, with no AI model involved. They support whole-workflow and agent node targets.

- Contains Text: passes when the output contains the given text.
- Exact Match: passes when the output is an exact match.
- JSON Similarity: passes when the JSON structures are similar.

### LLM judge

These evaluators use a large language model (LLM) to judge the output against the expected output. They support whole-workflow and agent node targets.

- LLM Judge Output: an AI model judges whether the output semantically matches the expected output.
- LLM Judge Strict JSON: an AI model judges a strict JSON structural match.
- LLM Judge Trajectory: an AI model judges whether the run followed the expected behavior, which is the steps taken during the run.
- LLM Judge Simulation: an AI model judges the path of the simulated run.

### Tool call

These evaluators check how an agent called its tools. They support agent node targets only.

- Tool Call Arguments: checks that tools were called with the correct arguments.
- Tool Call Count: checks the number of times each tool was called.
- Tool Call Order: checks that tools were called in the right sequence.
- Tool Call Output: checks that tool outputs match the expected values.

## Creating a dataset

The **Add** menu on the **Datasets** tab has three entries: **Add dataset**, **Import dataset**, and **Add data point via wizard**.

The **Add dataset** panel asks for a name and an optional description, an evaluation target, and, for a whole-process target, the entry point. It also lists the evaluators, each with a toggle: **LLM Judge Trajectory**, **Exact Match**, and **LLM Judge Evaluator**. A **Default simulations** section sets defaults that every data point inherits, and an individual data point can override them.

The **Add a data point for evaluation** wizard has four steps: **What to evaluate**, **Scenario input**, **Simulations**, and **Evaluation criteria**. The four steps are:

1. **What to evaluate**: the target, an optional data point name, which is generated when left blank, and the dataset to add the data point to.
2. **Scenario input**: the input that the trigger sends into the workflow, with one field for each trigger input, in a **JSON** or **Visual** view. The sparkle icon, **Generate with AI**, opens an **Instructions** box, and **Generate** creates the input from the instructions.
3. **Simulations**: overrides for the outputs of selected nodes during the evaluation run. Each node has a checkbox and a mock type, such as **Static mock**, the same option that [output mocking](output-mocking.md) offers on a node, with the output values that replace the node's real output. The step says the overrides use LLM instructions or rule-based simulations.
4. **Evaluation criteria**: the enabled evaluators and how each one scores this data point. For an LLM judge such as **LLM Judge Trajectory**, a text field describes the expected behavior. A **Start evaluation** checkbox runs the evaluation when the data point is added.

## Configuring an evaluator

The **Evaluators** tab lists the evaluators and has a **New evaluator** button. Every evaluator has a name, a description, and a target output key, which is either `*` for all workflow outputs or the output of one node, such as `httpRequest1`.

- **Exact Match** has a **Case Sensitive** toggle and a **Negated** toggle. When negated, the evaluation passes if the outputs do not match. Its default evaluation criteria hold the expected output, which is entered in a **JSON** or **Visual** view. If a target output key is set, the expected value is nested under that key.
- **LLM Judge Evaluator** has a model, such as `anthropic.claude-sonnet-5`, and a prompt that asks the model to score the similarity of the output and to explain the score. Its settings also include an **Advanced settings** section.

## Running an evaluation

The **Run evaluation** button opens the **Configure evaluation run** panel. The panel asks for the dataset, an optional filter by tags, and the evaluators to use. **Run settings** hold an optional run name, a batch size (10 by default), and a timeout (20 minutes by default). **Save & run** starts the run.

## Dataset lifecycle

A dataset is built around an evaluation target, data points, and evaluators. The target is either a single node or the whole workflow, entered through a trigger. Each data point supplies input values and the expected output, and each evaluator defines how the execution result is scored.

When a dataset runs, the target is executed for each data point and the per-evaluator scores appear in the **Eval runs** tab.

## Capturing a data point from a run

A data point does not have to be written by hand. The arrow next to the **Run evaluation** icon in the bottom toolbar opens **Add data point**, which starts the same wizard. The **Add to evaluations** button in the details of a debug run, and the flask icon on each row of the **Debug runs** list, capture that run, and its inputs and outputs become the data point's input and expected output. Capturing a run seeds a dataset from real behavior that has already been observed.

## Using evaluations in development

Evaluations are most useful in these situations:

- A workflow invokes AI agents, and the agent's output must meet quality expectations before you deploy.
- Regression coverage is needed: running evaluations after each change confirms that existing behavior has not been broken.
- Two alternative implementations are being compared by running evaluations on two versions of the same workflow.

## Evaluation results

Results are stored in the **Eval runs** tab, which shows "No evaluation runs yet" until a dataset has run, and they can be reviewed at any time. Each run is a row with its run number, the dataset, the status (for example **Completed**, or a progress count such as `0/1` while the run is in progress), the number of data points, the score of each evaluator as a percentage, the start time, and the duration. A search field, a filter, and a column selector sit above the table, and **Run evaluation** starts a new run. The **Actions** column of a run has a play button, **Start a new run from this one**, and a three-dot menu. Each run shows the per-evaluator scores, the individual data point results, and the actual and expected values for each data point.

## Practical example

A workflow uses an AI agent to classify customer support emails. Before deploying, a dataset with 10 data points is set up:

- 5 emails that should be classified as `"billing"`
- 3 emails that should be classified as `"technical"`
- 2 edge cases (ambiguous emails)

Each data point defines the expected `classification` output value. An **Exact match** evaluator on the classification scores each result, and the evaluation run shows which cases the agent got wrong before the workflow goes to production.

## Common mistakes

- Only running a dataset once: datasets are most useful after every significant change to a workflow that invokes an AI agent. Agent behavior can shift as models are updated even if the workflow has not changed.
- Writing data points only for the expected case: evaluations are most valuable for edge cases and failure modes. Strong data points include inputs that are ambiguous, malformed, or at the boundary of the expected range.
- Treating an all-pass score as a reason to stop testing: three passing data points are not confidence. Larger sets, especially ones that reflect real production inputs, provide stronger coverage.

## Next steps
- [Output mocking](output-mocking.md): fixed node outputs for debug runs.
- [Monitor runs](monitor-runs.md): run history and the trace of each run.
- [Autonomous Agent node](node-agent.md): the node that runs an AI agent.
- [Add an AI agent to a workflow](add-an-ai-agent.md): a build guide that runs an agent in a workflow.
- [Community](community.md): ask questions and share workflows with other builders on the UiPath Forum.