uip maestro flow eval

Syntax and options for `uip maestro flow eval`, which manages evaluators, evaluation sets, and data points locally and runs them in Studio Web.

uip maestro flow eval manages a Flow project's evaluation assets — evaluators (scoring logic), evaluation sets (named groups of test cases), and data points (individual test cases within a set) — and runs them remotely in Studio Web. All of this edits local JSON files in the Flow project except eval run, which talks to Studio Web.

Conceitos

• Evaluator — scoring logic (exact match, JSON similarity, contains, or an LLM judge) applied to a data point's actual output.
• Evaluation set — a named collection of data points plus the evaluators that score them. Can be scoped to the whole flow or to one agent node (--entry-point with an agent node id).
• Data point — one test case in a set: inputs, optional expected output, optional per-evaluator criteria.
• Simulation — a mocked or LLM-simulated response for one component (a node, or a child tool on an agent node), attached to a specific data point — lets you test a flow without calling the real dependency.
• Every local subcommand (evaluator, set, the bare data-point verbs, simulation) takes --path <path> — a Flow project directory, or a solution directory with exactly one Flow project. Defaults to ..

Resumo

uip maestro flow eval evaluator add    <name> --type <type> [--description <text>] [--target-key <key>] [--model <model>] [--prompt <text>] [--path <path>]
uip maestro flow eval evaluator list   [--path <path>]
uip maestro flow eval evaluator remove <id> [--path <path>]

uip maestro flow eval set add    <name> [--evaluators <refs>] [--entry-point <id>] [--path <path>]
uip maestro flow eval set list   [--path <path>]
uip maestro flow eval set remove <id> [--path <path>]

uip maestro flow eval add    <name> --set <name> [--inputs <json>] [--input-file <key=path>]... [--expected <json>] [--criteria <json>] [--search-text <text>] [--path <path>]
uip maestro flow eval list   --set <name> [--path <path>]
uip maestro flow eval remove <id> --set <name> [--path <path>]

uip maestro flow eval simulation add    <component-id> --set <name> --data-point <id> --strategy <Static|Llm> [options...] [--path <path>]
uip maestro flow eval simulation list   --set <name> --data-point <id> [--parent <component-id>] [--path <path>]
uip maestro flow eval simulation remove <component-id> --set <name> --data-point <id> [--parent <component-id>] [--path <path>]

uip maestro flow eval run start    --set <name> [--solution-id <id> | --project-id <id>] [--entry-point <entry>] [--folder-key <key>] [--debug-mode <mode>] [--wait [--timeout <seconds>]]
uip maestro flow eval run status   <evalSetRunId> --set <name> [--solution-id <id> | --project-id <id>]
uip maestro flow eval run results  <evalSetRunId> --set <name> [--only-failed] [--verbose] [--export-format <json|csv>]
uip maestro flow eval run list     --set <name> [--solution-id <id> | --project-id <id>]
uip maestro flow eval run compare  <evalSetRunId> --compare-to <id> --set <name>

Note the naming: data points are managed with bare eval add/list/remove (not eval evaluation add — there is no "evaluation" word in the actual command path, even though the source internally calls this concept "evaluation").

Honors global options. Exit codes follow the standard contract. eval run subcommands require uip login.

uip maestro flow eval evaluator

Manage evaluators — the scoring logic applied to data points.

uip maestro flow eval evaluator add

Opção	Required	Description
<name> (argument)	sim	Evaluator name.
--type <type>	sim	One of: exact-match, json-similarity, contains, llm-judge-output, llm-judge-strict-json, llm-judge-trajectory, llm-judge-trajectory-simulation.
--description <text>	não	Evaluator description.
--target-key <key>	não	Target output key to score. Default * (whole output).
--model <model>	não	LLM model — only meaningful for llm-judge-* types.
--prompt <text>	não	Custom LLM judge prompt — only meaningful for llm-judge-* types.
--path <path>	não	Flow project directory. Default ..

Formato dos dados: Code: "FlowEvalEvaluatorAdd", Data: { Name, Id, Type, File }.

uip maestro flow eval evaluator list

Data shape: Code: "FlowEvalEvaluatorList", Data is an array of { Name, Id, Type, TargetKey, File }.

uip maestro flow eval evaluator remove

Argument: <id> — evaluator ID, name, or file base name. Data shape: Code: "FlowEvalEvaluatorRemove".

uip maestro flow eval set

Manage evaluation sets.

uip maestro flow eval set add

Opção	Required	Description
<name> (argument)	sim	Evaluation set name.
--evaluators <refs>	não	Comma-separated evaluator IDs or file base names. Default: all evaluators in the project.
--entry-point <id>	não	Entry point node ID, stored as the set's selectedEntrypoint. Passing an agent node's ID creates a node-scoped set that evaluates just that agent in isolation.
--path <path>	não	Flow project directory.

Data shape: Code: "FlowEvalSetAdd", Data: { Name, Id, Evaluators, File, Scope, TargetNode? } — TargetNode is present only for a node-scoped set.

uip maestro flow eval set list

Data shape: Code: "FlowEvalSetList", Data is an array of { Name, Id, DataPoints, Evaluators, File, Scope, TargetNode? }.

uip maestro flow eval set remove

Argument: <id> — set ID, name, or file base name. Data shape: Code: "FlowEvalSetRemove".

uip maestro flow eval (data points)

Manage individual test cases within an evaluation set. Registered directly on eval, not under a data-point or evaluation sub-group.

uip maestro flow eval add

Opção	Required	Description
<name> (argument)	sim	Data point name.
--set <name>	sim	Nome ou ID do conjunto de avaliação.
--inputs <json>	no*	Input values as a JSON object.
--input-file <key=path>	no*	Attach a file as input <key>. Repeatable.
--expected <json>	não	Expected output as a JSON object.
--criteria <json>	não	Per-evaluator criteria, a JSON object keyed by evaluator ID.
--search-text <text>	não	Search text, for contains-type evaluators.
--path <path>	não	Flow project directory.

* At least one of --inputs/--input-file is required.

Formato dos dados: Code: "FlowEvalAdd", Data: { Status, Name, Id, Set }.

uip maestro flow eval list

Requires --set <name>. Data shape: Code: "FlowEvalList", Data is an array of { Name, Id, Inputs, Expected, Evaluators } (Inputs/Expected are stringified JSON; Expected is "-" when absent).

uip maestro flow eval remove

Arguments: <id> (data point ID or name). Requires --set <name>. Data shape: Code: "FlowEvalRemove".

uip maestro flow eval simulation

Mock or LLM-simulate a component's response for one data point — lets a data point exercise a flow without hitting the real dependency (an external API, a nested agent tool, and so on).

uip maestro flow eval simulation add

Opção	Required	Description
<component-id> (argument)	sim	Component ID to simulate.
--set <name>	sim	Nome ou ID do conjunto de avaliação.
--data-point <id>	sim	Data point name or ID.
--strategy <Static|Llm>	sim	Simulation strategy.
--component-type <type>	conditionally	Defaults to Node when --parent is passed; otherwise required.
--component-description <text>	não	Component description.
--simulation-instructions <text>	required with Llm	LLM simulation instructions.
--mock-value <json>	required with Static	Mock output value as JSON.
--parent <component-id>	não	Parent agent component ID. When set, this is added as a child tool simulation on that agent node instead of a top-level component simulation.
--path <path>	não	Flow project directory.

Data shape: Code: "FlowEvalSimulationAdd" (or "FlowEvalChildSimulationAdd" when --parent is set), Data: { ComponentId, ComponentType, Strategy, DataPoint, Set, Parent? }.

uip maestro flow eval simulation list

Requires --set/--data-point. --parent <component-id> lists child tool simulations on that agent node instead of top-level ones. Data shape: Code: "FlowEvalSimulationList" (or "FlowEvalChildSimulationList" with --parent).

uip maestro flow eval simulation remove

Argument: <component-id>. Requires --set/--data-point; --parent to target a child simulation. Data shape: Code: "FlowEvalSimulationRemove".

uip maestro flow eval run

Run an evaluation set remotely in Studio Web and inspect results. Every subcommand takes --set <name> (required), plus --solution-id <id> or --project-id <id> to pin the target Studio Web project (by default, read from the parent .uipx or SolutionStorage.json), and --path <path>.

uip maestro flow eval run start

Additional options: --entry-point <entry> (a Flow entry point path like /Main.bpmn#start, or a start node ID), --folder-key <key> (Orchestrator folder — uses the personal workspace if omitted), --debug-mode <mode> (Studio Web debug mode override), --wait (block until completion and print results), --timeout <seconds> (with --wait).

uip maestro flow eval run start --set "Regression Suite" --wait

Data shape: Code: "FlowEvalRunStarted", Data: { EvalSetRunId, EvalSetName, DataPoints, Evaluators, Status }. Without --wait, Instructions tells you the exact eval run status command to poll with.

uip maestro flow eval run status

Argument: <evalSetRunId>. Data shape: Code: "FlowEvalRunStatus", Data: { EvalSetRunId, Status, Score, Duration, EvaluatorScores } (Score is "-" if not yet scored).

uip maestro flow eval run results

Argument: <evalSetRunId>. Additional options: --only-failed (filter to failed/errored data points), --verbose (include evaluator justifications), --export-format <json|csv> (write to file instead of printing). Data shape: Code: "FlowEvalRunResults", Data is an array of per-data-point result rows.

uip maestro flow eval run list

No arguments beyond the shared target options. Data shape: Code: "FlowEvalRunList", Data is an array of past run summaries for the set.

uip maestro flow eval run compare

Arguments: <evalSetRunId> plus --compare-to <id> (required) — the second run to diff against. Data shape: Code: "FlowEvalRunComparison", Data is a structured diff between the two runs' scores and per-data-point outcomes.

Veja também

• uip maestro flow debug — smoke-test a flow interactively, as opposed to running a scored evaluation set
• uip maestro flow init, uip maestro flow node
• Visão geral do fluxo
• Opções globais, Códigos de saída