# uip maestro flow eval

> Syntax and options for `uip maestro flow eval`, which manages evaluators, evaluation sets, and data points locally and runs them in Studio Web.

`uip maestro flow eval` manages a Flow project's evaluation assets — **evaluators** (scoring logic), **evaluation sets** (named groups of test cases), and **data points** (individual test cases within a set) — and runs them remotely in Studio Web. All of this edits local JSON files in the Flow project except `eval run`, which talks to Studio Web.

## Concepts

- **Evaluator** — scoring logic (exact match, JSON similarity, contains, or an LLM judge) applied to a data point's actual output.
- **Evaluation set** — a named collection of data points plus the evaluators that score them. Can be scoped to the whole flow or to one agent node (`--entry-point` with an agent node id).
- **Data point** — one test case in a set: inputs, optional expected output, optional per-evaluator criteria.
- **Simulation** — a mocked or LLM-simulated response for one component (a node, or a child tool on an agent node), attached to a specific data point — lets you test a flow without calling the real dependency.
- Every local subcommand (`evaluator`, `set`, the bare data-point verbs, `simulation`) takes `--path <path>` — a Flow project directory, or a solution directory with exactly one Flow project. Defaults to `.`.

## Synopsis

```
uip maestro flow eval evaluator add    <name> --type <type> [--description <text>] [--target-key <key>] [--model <model>] [--prompt <text>] [--path <path>]
uip maestro flow eval evaluator list   [--path <path>]
uip maestro flow eval evaluator remove <id> [--path <path>]

uip maestro flow eval set add    <name> [--evaluators <refs>] [--entry-point <id>] [--path <path>]
uip maestro flow eval set list   [--path <path>]
uip maestro flow eval set remove <id> [--path <path>]

uip maestro flow eval add    <name> --set <name> [--inputs <json>] [--input-file <key=path>]... [--expected <json>] [--criteria <json>] [--search-text <text>] [--path <path>]
uip maestro flow eval list   --set <name> [--path <path>]
uip maestro flow eval remove <id> --set <name> [--path <path>]

uip maestro flow eval simulation add    <component-id> --set <name> --data-point <id> --strategy <Static|Llm> [options...] [--path <path>]
uip maestro flow eval simulation list   --set <name> --data-point <id> [--parent <component-id>] [--path <path>]
uip maestro flow eval simulation remove <component-id> --set <name> --data-point <id> [--parent <component-id>] [--path <path>]

uip maestro flow eval run start    --set <name> [--solution-id <id> | --project-id <id>] [--entry-point <entry>] [--folder-key <key>] [--debug-mode <mode>] [--wait [--timeout <seconds>]]
uip maestro flow eval run status   <evalSetRunId> --set <name> [--solution-id <id> | --project-id <id>]
uip maestro flow eval run results  <evalSetRunId> --set <name> [--only-failed] [--verbose] [--export-format <json|csv>]
uip maestro flow eval run list     --set <name> [--solution-id <id> | --project-id <id>]
uip maestro flow eval run compare  <evalSetRunId> --compare-to <id> --set <name>
```

Note the naming: data points are managed with bare `eval add`/`list`/`remove` (not `eval evaluation add` — there is no "evaluation" word in the actual command path, even though the source internally calls this concept "evaluation").

Honors [global options](./global-options.md). Exit codes follow the [standard contract](./exit-codes.md). `eval run` subcommands require `uip login`.

## uip maestro flow eval evaluator

Manage evaluators — the scoring logic applied to data points.

### uip maestro flow eval evaluator add

| Option | Required | Description |
|---|---|---|
| `<name>` (argument) | yes | Evaluator name. |
| `--type <type>` | yes | One of: `exact-match`, `json-similarity`, `contains`, `llm-judge-output`, `llm-judge-strict-json`, `llm-judge-trajectory`, `llm-judge-trajectory-simulation`. |
| `--description <text>` | no | Evaluator description. |
| `--target-key <key>` | no | Target output key to score. Default `*` (whole output). |
| `--model <model>` | no | LLM model — only meaningful for `llm-judge-*` types. |
| `--prompt <text>` | no | Custom LLM judge prompt — only meaningful for `llm-judge-*` types. |
| `--path <path>` | no | Flow project directory. Default `.`. |

Data shape: `Code: "FlowEvalEvaluatorAdd"`, `Data: { Name, Id, Type, File }`.

### uip maestro flow eval evaluator list

Data shape: `Code: "FlowEvalEvaluatorList"`, `Data` is an array of `{ Name, Id, Type, TargetKey, File }`.

### uip maestro flow eval evaluator remove

Argument: `<id>` — evaluator ID, name, or file base name. Data shape: `Code: "FlowEvalEvaluatorRemove"`.

## uip maestro flow eval set

Manage evaluation sets.

### uip maestro flow eval set add

| Option | Required | Description |
|---|---|---|
| `<name>` (argument) | yes | Evaluation set name. |
| `--evaluators <refs>` | no | Comma-separated evaluator IDs or file base names. Default: all evaluators in the project. |
| `--entry-point <id>` | no | Entry point node ID, stored as the set's `selectedEntrypoint`. Passing an **agent node's** ID creates a node-scoped set that evaluates just that agent in isolation. |
| `--path <path>` | no | Flow project directory. |

Data shape: `Code: "FlowEvalSetAdd"`, `Data: { Name, Id, Evaluators, File, Scope, TargetNode? }` — `TargetNode` is present only for a node-scoped set.

### uip maestro flow eval set list

Data shape: `Code: "FlowEvalSetList"`, `Data` is an array of `{ Name, Id, DataPoints, Evaluators, File, Scope, TargetNode? }`.

### uip maestro flow eval set remove

Argument: `<id>` — set ID, name, or file base name. Data shape: `Code: "FlowEvalSetRemove"`.

## uip maestro flow eval (data points)

Manage individual test cases within an evaluation set. Registered directly on `eval`, not under a `data-point` or `evaluation` sub-group.

### uip maestro flow eval add

| Option | Required | Description |
|---|---|---|
| `<name>` (argument) | yes | Data point name. |
| `--set <name>` | yes | Evaluation set name or ID. |
| `--inputs <json>` | no* | Input values as a JSON object. |
| `--input-file <key=path>` | no* | Attach a file as input `<key>`. Repeatable. |
| `--expected <json>` | no | Expected output as a JSON object. |
| `--criteria <json>` | no | Per-evaluator criteria, a JSON object keyed by evaluator ID. |
| `--search-text <text>` | no | Search text, for `contains`-type evaluators. |
| `--path <path>` | no | Flow project directory. |

\* At least one of `--inputs`/`--input-file` is required.

Data shape: `Code: "FlowEvalAdd"`, `Data: { Status, Name, Id, Set }`.

### uip maestro flow eval list

Requires `--set <name>`. Data shape: `Code: "FlowEvalList"`, `Data` is an array of `{ Name, Id, Inputs, Expected, Evaluators }` (`Inputs`/`Expected` are stringified JSON; `Expected` is `"-"` when absent).

### uip maestro flow eval remove

Arguments: `<id>` (data point ID or name). Requires `--set <name>`. Data shape: `Code: "FlowEvalRemove"`.

## uip maestro flow eval simulation

Mock or LLM-simulate a component's response for one data point — lets a data point exercise a flow without hitting the real dependency (an external API, a nested agent tool, and so on).

### uip maestro flow eval simulation add

| Option | Required | Description |
|---|---|---|
| `<component-id>` (argument) | yes | Component ID to simulate. |
| `--set <name>` | yes | Evaluation set name or ID. |
| `--data-point <id>` | yes | Data point name or ID. |
| `--strategy <Static\|Llm>` | yes | Simulation strategy. |
| `--component-type <type>` | conditionally | Defaults to `Node` when `--parent` is passed; otherwise required. |
| `--component-description <text>` | no | Component description. |
| `--simulation-instructions <text>` | required with `Llm` | LLM simulation instructions. |
| `--mock-value <json>` | required with `Static` | Mock output value as JSON. |
| `--parent <component-id>` | no | Parent agent component ID. When set, this is added as a **child tool simulation** on that agent node instead of a top-level component simulation. |
| `--path <path>` | no | Flow project directory. |

Data shape: `Code: "FlowEvalSimulationAdd"` (or `"FlowEvalChildSimulationAdd"` when `--parent` is set), `Data: { ComponentId, ComponentType, Strategy, DataPoint, Set, Parent? }`.

### uip maestro flow eval simulation list

Requires `--set`/`--data-point`. `--parent <component-id>` lists child tool simulations on that agent node instead of top-level ones. Data shape: `Code: "FlowEvalSimulationList"` (or `"FlowEvalChildSimulationList"` with `--parent`).

### uip maestro flow eval simulation remove

Argument: `<component-id>`. Requires `--set`/`--data-point`; `--parent` to target a child simulation. Data shape: `Code: "FlowEvalSimulationRemove"`.

## uip maestro flow eval run

Run an evaluation set remotely in Studio Web and inspect results. Every subcommand takes `--set <name>` (required), plus `--solution-id <id>` or `--project-id <id>` to pin the target Studio Web project (by default, read from the parent `.uipx` or `SolutionStorage.json`), and `--path <path>`.

### uip maestro flow eval run start

Additional options: `--entry-point <entry>` (a Flow entry point path like `/Main.bpmn#start`, or a start node ID), `--folder-key <key>` (Orchestrator folder — uses the personal workspace if omitted), `--debug-mode <mode>` (Studio Web debug mode override), `--wait` (block until completion and print results), `--timeout <seconds>` (with `--wait`).

```bash
uip maestro flow eval run start --set "Regression Suite" --wait
```

Data shape: `Code: "FlowEvalRunStarted"`, `Data: { EvalSetRunId, EvalSetName, DataPoints, Evaluators, Status }`. Without `--wait`, `Instructions` tells you the exact `eval run status` command to poll with.

### uip maestro flow eval run status

Argument: `<evalSetRunId>`. Data shape: `Code: "FlowEvalRunStatus"`, `Data: { EvalSetRunId, Status, Score, Duration, EvaluatorScores }` (`Score` is `"-"` if not yet scored).

### uip maestro flow eval run results

Argument: `<evalSetRunId>`. Additional options: `--only-failed` (filter to failed/errored data points), `--verbose` (include evaluator justifications), `--export-format <json|csv>` (write to file instead of printing). Data shape: `Code: "FlowEvalRunResults"`, `Data` is an array of per-data-point result rows.

### uip maestro flow eval run list

No arguments beyond the shared target options. Data shape: `Code: "FlowEvalRunList"`, `Data` is an array of past run summaries for the set.

### uip maestro flow eval run compare

Arguments: `<evalSetRunId>` plus `--compare-to <id>` (required) — the second run to diff against. Data shape: `Code: "FlowEvalRunComparison"`, `Data` is a structured diff between the two runs' scores and per-data-point outcomes.

## See also

- [`uip maestro flow debug`](./uip-maestro-flow-debug.md) — smoke-test a flow interactively, as opposed to running a scored evaluation set
- [`uip maestro flow init`](./uip-maestro-flow-init.md), [`uip maestro flow node`](./uip-maestro-flow-node-edge.md)
- [Flow overview](./uip-maestro-flow.md)
- [Global options](./global-options.md), [Exit codes](./exit-codes.md)