uip eval

Syntax and options for `uip eval`, which runs runtime evaluations against Orchestrator packages and manages evaluators, eval sets, evaluations, and schedules.

uip eval runs runtime evaluations against a deployed Orchestrator package: submit real inputs through a process, score the outputs with one or more evaluators, and inspect the results. It also manages the reusable building blocks behind that workflow — evaluators (scoring logic), eval sets (named collections that group evaluations), evaluations (individual saved test-case definitions — name, inputs, expected output/behavior), and recurring schedules that trigger runs automatically.

This resource spans four pages, split by sub-domain:

• This page — concepts, and execute-and-evaluate / run (submit a run, then list/inspect its results).
• schedule — create/list/get/update/pause/resume/delete recurring eval runs.
• evaluator — manage the scoring logic (e.g. LLM-judge evaluators) a run applies to its outputs.
• eval-set and evaluation — manage curated item collections and individual evaluation definitions.

uip eval ships as its own standalone package (@uipath/eval-tool, command prefix eval) — it is not a sub-command of uip or, even though every verb here operates against an Orchestrator process/folder. For the shared Orchestrator conventions (auth, folder scoping, key types, output envelope) that this tool's --process-key/--folder-key/--tenant options follow, see the uip or overview.

概念

• Eval item — one test case passed inline to execute-and-evaluate: an id, a name, and an inputs object matching the process's expected input schema. A run submits an array of these directly on the command line, whether or not it also references a saved eval set.
• Evaluation — the saved counterpart to an eval item: a persisted test-case definition (name, inputs, and optionally an expected output/behavior) that belongs to an eval set. Manage these with eval evaluation.
• Evaluator — scoring logic applied to each item's output, identified by an evaluatorTypeId (for example uipath-llm-judge-output-semantic-similarity) plus an evaluatorConfig object. Manage saved evaluators with eval evaluator; a run can also pass evaluator configs inline via --evaluators.
• Eval set — a named, reusable collection of evaluations, managed with eval eval-set. execute-and-evaluate can reference one by --eval-set-id for tracking/grouping purposes, or omit it to run ad hoc (defaults to the zero GUID 00000000-0000-0000-0000-000000000000) — either way, the actual items scored in a given run are whatever --items contains on that call.
• Eval set run vs. eval run — submitting execute-and-evaluate creates one eval set run (the overall batch, identified by EvalSetRunId), which fans out into one eval run per item (run results lists these individually, each with its own Status/Result).
• Folder scoping — execute-and-evaluate and run resolve a folder from --folder-key, or default to your personal workspace when omitted (the CLI looks it up automatically). This differs from most uip or verbs, which require an explicit folder selector — see the overview for the general convention.

大纲

uip eval execute-and-evaluate --process-key <guid> --workload-id <guid> --items <json> --evaluators <json> [--eval-set-id <guid>] [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval run list    --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <name>]
uip eval run get     <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval run results <evalSetRunId> --process-key <guid> [--tenant <name>]

execute-and-evaluate has an alias: run-offline-evals.

uip eval execute-and-evaluate

Submit a runtime eval run for an Orchestrator package: send a batch of items through the process identified by --process-key, score each output with the given evaluators, and return the EvalSetRunId to poll with run get/run results.

选项

短	长	值	默认	描述
—	--process-key	GUID	—	Required. Process key. Use uip or processes list to find available keys.
—	--workload-id	GUID	—	Required. Workload ID.
—	--items	JSON array	—	Required. Eval items, each with id (string), name (string), and inputs (object). Validated client-side before the call — a missing/malformed field on any item fails fast, naming the item index.
—	--evaluators	JSON array	—	Required. Evaluator configs, each with an id, evaluatorTypeId, and evaluatorConfig.
—	--eval-set-id	GUID	zero GUID	Reference a saved eval set. Omit for an ad hoc run.
—	--batch-size	数字	5	Max concurrent evaluation pipelines.
—	--folder-key	GUID	personal workspace	Folder to run in. Use uip or folders list to find available keys.
—	--tenant	name	会话默认值	UiPath tenant name. Plain option — see Concepts.

示例

# Submit a run with an ad hoc item and an inline evaluator
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]'

# Reference a saved eval set and a specific folder
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]' \
  --eval-set-id a1b2c3d4-0000-0000-0000-000000000001 \
  --folder-key a9f3b2c1-7d4e-4a8b-9c2f-5e1d3b6a8f7e

数据形状（--输出 json）

{
  "Code": "EvalRunSubmitted",
  "Data": {
    "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09",
    "Folder": "[e-mail]'s workspace",
    "EvalSetId": "00000000-0000-0000-0000-000000000000",
    "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04"
  }
}

Folder echoes the resolved personal-workspace name when --folder-key was omitted, or the raw key you passed otherwise.

uip eval run

Inspect eval set runs created by execute-and-evaluate, scoped by --process-key.

uip eval run list

List eval set runs for a process.

选项

短	长	值	默认	描述
—	--process-key	GUID	—	Required. Process key (deployment GUID).
—	--limit	数字	50	Max runs to return.
—	--offset	数字	0	Number of runs to skip.
—	--tenant	name	会话默认值	UiPath tenant name.

示例

uip eval run list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

数据形状（--输出 json）

{
  "Code": "EvalSetRunList",
  "Data": [
    {
      "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Status": "completed",
      "Score": 0.86,
      "EvalsExecuted": 5,
      "Duration": "42.5s",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "returned": 1, "limit": 50, "offset": 0 }
}

Score and Duration are "-" when not yet available (run still in progress). Duration is formatted as Nms under one second, N.Ns otherwise.

uip eval run get

Get details of one eval set run.

参数

名称	必填	用途
<evalSetRunId>	是	Eval set run ID (GUID), from run list or execute-and-evaluate's output.

选项

短	长	值	默认	描述
—	--process-key	GUID	—	Required. Process key (deployment GUID).
—	--tenant	name	会话默认值	UiPath tenant name.

示例

uip eval run get a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

数据形状（--输出 json）

{
  "Code": "EvalSetRunDetails",
  "Data": {
    "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
    "Status": "completed",
    "Score": 0.86,
    "EvalsExecuted": 5,
    "Duration": "42.5s"
  }
}

Same shape as one run list entry (minus EvalSetId/CreatedAt), fetched by ID instead of listed.

uip eval run results

View per-item eval run results for an eval set run — one entry per item submitted in the original execute-and-evaluate call.

参数

名称	必填	用途
<evalSetRunId>	是	Eval set run ID (GUID).

选项

短	长	值	默认	描述
—	--process-key	GUID	—	Required. Process key (deployment GUID).
—	--tenant	name	会话默认值	UiPath tenant name.

示例

uip eval run results a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

数据形状（--输出 json）

{
  "Code": "EvalRunResults",
  "Data": [
    {
      "EvalRunId": "e1e2e3e4-0000-0000-0000-000000000001",
      "DataPoint": "Test Case 1",
      "Status": "completed",
      "Result": {}
    }
  ]
}

DataPoint is the originating item's name (falls back to the eval run's own ID if the snapshot has none). Result is the evaluator's raw scoring payload — shape depends on which evaluator ran.

相关内容

• schedule — automate execute-and-evaluate on a recurring schedule.
• evaluator — manage saved scoring logic.
• eval-set and evaluation — manage saved item collections and evaluation definitions.

另请参阅

• uip or overview — the tool whose processes and folders this one operates against; shared conventions for --process-key/--folder-key/--tenant.
• 全局选项
• 退出代码