uip eval

Syntax and options for `uip eval`, which runs runtime evaluations against Orchestrator packages and manages evaluators, eval sets, evaluations, and schedules.

uip eval runs runtime evaluations against a deployed Orchestrator package: submit real inputs through a process, score the outputs with one or more evaluators, and inspect the results. It also manages the reusable building blocks behind that workflow — evaluators (scoring logic), eval sets (named collections that group evaluations), evaluations (individual saved test-case definitions — name, inputs, expected output/behavior), and recurring schedules that trigger runs automatically.

This resource spans four pages, split by sub-domain:

• This page — concepts, and execute-and-evaluate / run (submit a run, then list/inspect its results).
• schedule — create/list/get/update/pause/resume/delete recurring eval runs.
• evaluator — manage the scoring logic (e.g. LLM-judge evaluators) a run applies to its outputs.
• eval-set and evaluation — manage curated item collections and individual evaluation definitions.

uip eval ships as its own standalone package (@uipath/eval-tool, command prefix eval) — it is not a sub-command of uip or, even though every verb here operates against an Orchestrator process/folder. For the shared Orchestrator conventions (auth, folder scoping, key types, output envelope) that this tool's --process-key/--folder-key/--tenant options follow, see the uip or overview.

Concepts

• Eval item — one test case passed inline to execute-and-evaluate: an id, a name, and an inputs object matching the process's expected input schema. A run submits an array of these directly on the command line, whether or not it also references a saved eval set.
• Evaluation — the saved counterpart to an eval item: a persisted test-case definition (name, inputs, and optionally an expected output/behavior) that belongs to an eval set. Manage these with eval evaluation.
• Evaluator — scoring logic applied to each item's output, identified by an evaluatorTypeId (for example uipath-llm-judge-output-semantic-similarity) plus an evaluatorConfig object. Manage saved evaluators with eval evaluator; a run can also pass evaluator configs inline via --evaluators.
• Eval set — a named, reusable collection of evaluations, managed with eval eval-set. execute-and-evaluate can reference one by --eval-set-id for tracking/grouping purposes, or omit it to run ad hoc (defaults to the zero GUID 00000000-0000-0000-0000-000000000000) — either way, the actual items scored in a given run are whatever --items contains on that call.
• Eval set run vs. eval run — submitting execute-and-evaluate creates one eval set run (the overall batch, identified by EvalSetRunId), which fans out into one eval run per item (run results lists these individually, each with its own Status/Result).
• Folder scoping — execute-and-evaluate and run resolve a folder from --folder-key, or default to your personal workspace when omitted (the CLI looks it up automatically). This differs from most uip or verbs, which require an explicit folder selector — see the overview for the general convention.

Synopsis

uip eval execute-and-evaluate --process-key <guid> --workload-id <guid> --items <json> --evaluators <json> [--eval-set-id <guid>] [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval run list    --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <name>]
uip eval run get     <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval run results <evalSetRunId> --process-key <guid> [--tenant <name>]

execute-and-evaluate has an alias: run-offline-evals.

uip eval execute-and-evaluate

Submit a runtime eval run for an Orchestrator package: send a batch of items through the process identified by --process-key, score each output with the given evaluators, and return the EvalSetRunId to poll with run get/run results.

Options

Short	Long	Value	Default	Description
—	--process-key	GUID	—	Required. Process key. Use uip or processes list to find available keys.
—	--workload-id	GUID	—	Required. Workload ID.
—	--items	JSON array	—	Required. Eval items, each with id (string), name (string), and inputs (object). Validated client-side before the call — a missing/malformed field on any item fails fast, naming the item index.
—	--evaluators	JSON array	—	Required. Evaluator configs, each with an id, evaluatorTypeId, and evaluatorConfig.
—	--eval-set-id	GUID	zero GUID	Reference a saved eval set. Omit for an ad hoc run.
—	--batch-size	number	5	Max concurrent evaluation pipelines.
—	--folder-key	GUID	personal workspace	Folder to run in. Use uip or folders list to find available keys.
—	--tenant	name	session default	UiPath tenant name. Plain option — see Concepts.

Examples

# Submit a run with an ad hoc item and an inline evaluator
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]'

# Reference a saved eval set and a specific folder
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]' \
  --eval-set-id a1b2c3d4-0000-0000-0000-000000000001 \
  --folder-key a9f3b2c1-7d4e-4a8b-9c2f-5e1d3b6a8f7e

Data shape (--output json)

{
  "Code": "EvalRunSubmitted",
  "Data": {
    "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09",
    "Folder": "[e-mail]'s workspace",
    "EvalSetId": "00000000-0000-0000-0000-000000000000",
    "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04"
  }
}

Folder echoes the resolved personal-workspace name when --folder-key was omitted, or the raw key you passed otherwise.

uip eval run

Inspect eval set runs created by execute-and-evaluate, scoped by --process-key.

uip eval run list

List eval set runs for a process.

Options

Short	Long	Value	Default	Description
—	--process-key	GUID	—	Required. Process key (deployment GUID).
—	--limit	number	50	Max runs to return.
—	--offset	number	0	Number of runs to skip.
—	--tenant	name	session default	UiPath tenant name.

Example

uip eval run list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalSetRunList",
  "Data": [
    {
      "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Status": "completed",
      "Score": 0.86,
      "EvalsExecuted": 5,
      "Duration": "42.5s",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "returned": 1, "limit": 50, "offset": 0 }
}

Score and Duration are "-" when not yet available (run still in progress). Duration is formatted as Nms under one second, N.Ns otherwise.

uip eval run get

Get details of one eval set run.

Arguments

Name	Required	Purpose
<evalSetRunId>	yes	Eval set run ID (GUID), from run list or execute-and-evaluate's output.

Options

Short	Long	Value	Default	Description
—	--process-key	GUID	—	Required. Process key (deployment GUID).
—	--tenant	name	session default	UiPath tenant name.

Example

uip eval run get a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalSetRunDetails",
  "Data": {
    "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
    "Status": "completed",
    "Score": 0.86,
    "EvalsExecuted": 5,
    "Duration": "42.5s"
  }
}

Same shape as one run list entry (minus EvalSetId/CreatedAt), fetched by ID instead of listed.

uip eval run results

View per-item eval run results for an eval set run — one entry per item submitted in the original execute-and-evaluate call.

Arguments

Name	Required	Purpose
<evalSetRunId>	yes	Eval set run ID (GUID).

Options

Short	Long	Value	Default	Description
—	--process-key	GUID	—	Required. Process key (deployment GUID).
—	--tenant	name	session default	UiPath tenant name.

Example

uip eval run results a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalRunResults",
  "Data": [
    {
      "EvalRunId": "e1e2e3e4-0000-0000-0000-000000000001",
      "DataPoint": "Test Case 1",
      "Status": "completed",
      "Result": {}
    }
  ]
}

DataPoint is the originating item's name (falls back to the eval run's own ID if the snapshot has none). Result is the evaluator's raw scoring payload — shape depends on which evaluator ran.

Related

• schedule — automate execute-and-evaluate on a recurring schedule.
• evaluator — manage saved scoring logic.
• eval-set and evaluation — manage saved item collections and evaluation definitions.

See also

• uip or overview — the tool whose processes and folders this one operates against; shared conventions for --process-key/--folder-key/--tenant.
• Global options
• Exit codes