# uip eval

> Syntax and options for `uip eval`, which runs runtime evaluations against Orchestrator packages and manages evaluators, eval sets, evaluations, and schedules.

`uip eval` runs runtime evaluations against a deployed Orchestrator package: submit real inputs through a process, score the outputs with one or more evaluators, and inspect the results. It also manages the reusable building blocks behind that workflow — evaluators (scoring logic), eval sets (named collections that group evaluations), evaluations (individual saved test-case definitions — name, inputs, expected output/behavior), and recurring schedules that trigger runs automatically.

This resource spans four pages, split by sub-domain:

- **This page** — concepts, and `execute-and-evaluate` / `run` (submit a run, then list/inspect its results).
- [`schedule`](./uip-eval-schedule.md) — create/list/get/update/pause/resume/delete recurring eval runs.
- [`evaluator`](./uip-eval-evaluators.md) — manage the scoring logic (e.g. LLM-judge evaluators) a run applies to its outputs.
- [`eval-set` and `evaluation`](./uip-eval-sets-evaluations.md) — manage curated item collections and individual evaluation definitions.

`uip eval` ships as its own standalone package (`@uipath/eval-tool`, command prefix `eval`) — it is not a sub-command of [`uip or`](./uip-or.md), even though every verb here operates against an Orchestrator process/folder. For the shared Orchestrator conventions (auth, folder scoping, key types, output envelope) that this tool's `--process-key`/`--folder-key`/`--tenant` options follow, see the [`uip or` overview](./uip-or.md).

## Concepts

- **Eval item** — one test case passed inline to `execute-and-evaluate`: an `id`, a `name`, and an `inputs` object matching the process's expected input schema. A run submits an array of these directly on the command line, whether or not it also references a saved eval set.
- **Evaluation** — the saved counterpart to an eval item: a persisted test-case definition (name, inputs, and optionally an expected output/behavior) that belongs to an eval set. Manage these with [`eval evaluation`](./uip-eval-sets-evaluations.md).
- **Evaluator** — scoring logic applied to each item's output, identified by an `evaluatorTypeId` (for example `uipath-llm-judge-output-semantic-similarity`) plus an `evaluatorConfig` object. Manage saved evaluators with [`eval evaluator`](./uip-eval-evaluators.md); a run can also pass evaluator configs inline via `--evaluators`.
- **Eval set** — a named, reusable collection of evaluations, managed with [`eval eval-set`](./uip-eval-sets-evaluations.md). `execute-and-evaluate` can reference one by `--eval-set-id` for tracking/grouping purposes, or omit it to run ad hoc (defaults to the zero GUID `00000000-0000-0000-0000-000000000000`) — either way, the actual items scored in a given run are whatever `--items` contains on that call.
- **Eval set run vs. eval run** — submitting `execute-and-evaluate` creates one **eval set run** (the overall batch, identified by `EvalSetRunId`), which fans out into one **eval run** per item (`run results` lists these individually, each with its own `Status`/`Result`).
- **Folder scoping** — `execute-and-evaluate` and `run` resolve a folder from `--folder-key`, or default to your **personal workspace** when omitted (the CLI looks it up automatically). This differs from most `uip or` verbs, which require an explicit folder selector — see the [overview](./uip-or.md) for the general convention.
## Synopsis

```
uip eval execute-and-evaluate --process-key <guid> --workload-id <guid> --items <json> --evaluators <json> [--eval-set-id <guid>] [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval run list    --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <name>]
uip eval run get     <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval run results <evalSetRunId> --process-key <guid> [--tenant <name>]
```

`execute-and-evaluate` has an alias: `run-offline-evals`.

## uip eval execute-and-evaluate

Submit a runtime eval run for an Orchestrator package: send a batch of items through the process identified by `--process-key`, score each output with the given evaluators, and return the `EvalSetRunId` to poll with `run get`/`run results`.

### Options

| Short | Long | Value | Default | Description |
|---|---|---|---|---|
| — | `--process-key` | GUID | — | **Required.** Process key. Use `uip or processes list` to find available keys. |
| — | `--workload-id` | GUID | — | **Required.** Workload ID. |
| — | `--items` | JSON array | — | **Required.** Eval items, each with `id` (string), `name` (string), and `inputs` (object). Validated client-side before the call — a missing/malformed field on any item fails fast, naming the item index. |
| — | `--evaluators` | JSON array | — | **Required.** Evaluator configs, each with an `id`, `evaluatorTypeId`, and `evaluatorConfig`. |
| — | `--eval-set-id` | GUID | zero GUID | Reference a saved eval set. Omit for an ad hoc run. |
| — | `--batch-size` | number | `5` | Max concurrent evaluation pipelines. |
| — | `--folder-key` | GUID | personal workspace | Folder to run in. Use `uip or folders list` to find available keys. |
| — | `--tenant` | name | session default | UiPath tenant name. Plain option — see [Concepts](#concepts). |

### Examples

```bash
# Submit a run with an ad hoc item and an inline evaluator
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]'
```

```bash
# Reference a saved eval set and a specific folder
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]' \
  --eval-set-id a1b2c3d4-0000-0000-0000-000000000001 \
  --folder-key a9f3b2c1-7d4e-4a8b-9c2f-5e1d3b6a8f7e
```

### Data shape (--output json)

```json
{
  "Code": "EvalRunSubmitted",
  "Data": {
    "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09",
    "Folder": "user@uipath.com's workspace",
    "EvalSetId": "00000000-0000-0000-0000-000000000000",
    "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04"
  }
}
```

`Folder` echoes the resolved personal-workspace name when `--folder-key` was omitted, or the raw key you passed otherwise.

## uip eval run

Inspect eval set runs created by `execute-and-evaluate`, scoped by `--process-key`.

### uip eval run list

List eval set runs for a process.

#### Options

| Short | Long | Value | Default | Description |
|---|---|---|---|---|
| — | `--process-key` | GUID | — | **Required.** Process key (deployment GUID). |
| — | `--limit` | number | `50` | Max runs to return. |
| — | `--offset` | number | `0` | Number of runs to skip. |
| — | `--tenant` | name | session default | UiPath tenant name. |

#### Example

```bash
uip eval run list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
```

#### Data shape (--output json)

```json
{
  "Code": "EvalSetRunList",
  "Data": [
    {
      "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Status": "completed",
      "Score": 0.86,
      "EvalsExecuted": 5,
      "Duration": "42.5s",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "returned": 1, "limit": 50, "offset": 0 }
}
```

`Score` and `Duration` are `"-"` when not yet available (run still in progress). `Duration` is formatted as `Nms` under one second, `N.Ns` otherwise.

### uip eval run get

Get details of one eval set run.

#### Arguments

| Name | Required | Purpose |
|---|---|---|
| `<evalSetRunId>` | yes | Eval set run ID (GUID), from `run list` or `execute-and-evaluate`'s output. |

#### Options

| Short | Long | Value | Default | Description |
|---|---|---|---|---|
| — | `--process-key` | GUID | — | **Required.** Process key (deployment GUID). |
| — | `--tenant` | name | session default | UiPath tenant name. |

#### Example

```bash
uip eval run get a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
```

#### Data shape (--output json)

```json
{
  "Code": "EvalSetRunDetails",
  "Data": {
    "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
    "Status": "completed",
    "Score": 0.86,
    "EvalsExecuted": 5,
    "Duration": "42.5s"
  }
}
```

Same shape as one `run list` entry (minus `EvalSetId`/`CreatedAt`), fetched by ID instead of listed.

### uip eval run results

View per-item eval run results for an eval set run — one entry per item submitted in the original `execute-and-evaluate` call.

#### Arguments

| Name | Required | Purpose |
|---|---|---|
| `<evalSetRunId>` | yes | Eval set run ID (GUID). |

#### Options

| Short | Long | Value | Default | Description |
|---|---|---|---|---|
| — | `--process-key` | GUID | — | **Required.** Process key (deployment GUID). |
| — | `--tenant` | name | session default | UiPath tenant name. |

#### Example

```bash
uip eval run results a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
```

#### Data shape (--output json)

```json
{
  "Code": "EvalRunResults",
  "Data": [
    {
      "EvalRunId": "e1e2e3e4-0000-0000-0000-000000000001",
      "DataPoint": "Test Case 1",
      "Status": "completed",
      "Result": {}
    }
  ]
}
```

`DataPoint` is the originating item's `name` (falls back to the eval run's own ID if the snapshot has none). `Result` is the evaluator's raw scoring payload — shape depends on which evaluator ran.

## Related

- [`schedule`](./uip-eval-schedule.md) — automate `execute-and-evaluate` on a recurring schedule.
- [`evaluator`](./uip-eval-evaluators.md) — manage saved scoring logic.
- [`eval-set` and `evaluation`](./uip-eval-sets-evaluations.md) — manage saved item collections and evaluation definitions.

## See also

- [`uip or` overview](./uip-or.md) — the tool whose processes and folders this one operates against; shared conventions for `--process-key`/`--folder-key`/`--tenant`.
- [Global options](./global-options.md)
- [Exit codes](./exit-codes.md)