Evaluations (Preview)

Automated quality checks for Maestro BPMN workflows, with evaluation sets, deterministic and LLM-judge evaluators, and node output simulations.

Evaluations let you define automated quality checks for a Maestro BPMN workflow. You group expected outcomes into an evaluation set, and each execution is scored against the evaluators you attach.

Evaluations use the same evaluation framework already available for AI agents and Maestro Flow, so workflows with non-deterministic steps can be verified before they ship.

概念

• Dataset — a collection of data points that target the same subject. Running the set scores the workflow against every data point at once.
• Data point — a single case: a set of input values and the expected output for those inputs. You can author a data point by hand, or capture one from an actual run.
• Evaluator — a grader that compares actual output against expected output, or the trajectory against an expected trajectory, and produces a score.
• 分数— 评估程序为数据点报告的结果。

BPMN-level evaluation sets

An evaluation set targets the whole Maestro BPMN workflow. It is pinned to a trigger entry point, and each data point supplies the trigger's inputs and the workflow's expected output.

Node-level evaluation sets pin an evaluation to a single node, so that each data point supplies that node's inputs and its expected output. Node-level evaluations are not part of this public preview.

评估者

Maestro BPMN evaluators fall into two families.

确定性

These evaluators compare output to expected output exactly, with no model in the loop.

• 包含— 检查输出是否包含预期子字符串。
• 精确匹配— 检查输出是否等于预期值。
• JSON 相似度— 比较输出和预期的 JSON 输出。

LLM 评判者

These evaluators use a model to judge the output against the expected output.

• 输出语义相似度— 判断输出的含义是否与预期输出相同。
• 输出 JSON 严格相似度— 对照预期 JSON 判断 JSON 输出。
• 轨迹相似度— 根据预期轨迹判断智能体采取的步数。
• 轨迹模拟— 判断模拟轨迹。

从运行中捕获数据点

You don't have to author every data point by hand. You can build one by importing an actual execution: when you capture a debug run, its inputs and outputs become the data point's input and expected output.

Capturing a run is a fast way to seed an evaluation set from real behavior you have already observed.

When to use evaluations

在以下情况下，评估最有用：

• You are building Maestro BPMN workflows that invoke AI agents and need to verify that the agent's output meets quality expectations before publishing.
• You need regression coverage, and running evaluations after each change confirms that existing behavior has not been broken.
• 您需要通过对同一工作流的两个版本运行评估来比较替代实施方案。

评估结果

Results are stored in the evaluation history and can be reviewed at any time. They are visible to the users who run them, and can be exported as either JSON or CSV. Individual traces inside an evaluation run can also be exported.

The evaluation runs history tab shows, for each run:

• Per-evaluator scores, together with the grader's justification
• Individual data point results, with actual and expected values side by side
• The full trace: what triggered the run, the path it took, per-node output, and timing

The trace and the grader justification indicate what needs to change for the workflow to run correctly on each case.

Example: classifying support emails

A Maestro BPMN workflow uses an AI agent to classify customer support emails. Before publishing, you set up an evaluation set with 10 data points:

• 5 emails classified as "billing"
• 3 emails classified as "technical"
• 2 ambiguous emails, treated as edge cases

Each data point defines the expected classification output value, or a triggered escalation to a human for the ambiguous emails. An Exact match evaluator on the classification scores each result for "billing" or "technical", and a trajectory evaluator checks the human escalation. Each run shows which cases the agent got wrong before the workflow goes to production.

Node simulations

Simulations override selected node outputs during an evaluation run, so you can test downstream logic without executing a node's real action. This is useful when the real action is slow, expensive, or depends on external systems that are not available during design.

When a simulation is enabled on a node, the run skips that node's actual execution and uses the mocked output values directly. All downstream nodes receive the mocked values as if the node had run normally.

Simulations apply only during test runs. They are never active in published, production workflows.

Simulation strategies

Each simulated node uses one of two strategies:

• Static mock — a rule-based simulation that always returns the value you define.
• Generated mock — an LLM-generated response, produced dynamically each time the evaluation runs.

Setting up a simulation

Configure a simulation on any node whose real output you want to replace during an evaluation run. The following sequence shows the full flow, from selecting the node through entering the simulated output.

1. Select the node you want to simulate on the canvas.
2. In the configuration panel, open Add simulation.
3. Choose the simulation strategy for the node.
4. Enter the simulated output for the strategy you chose:
   1. For Static mock, enter the output values, either by editing the JSON manually or by generating it with the prompt feature. Use valid JSON that matches the node's output schema.
   2. For Generated mock, enter the prompt the model uses to generate a tool response on each evaluation run.

Result: The node is simulated. On the next test run, Maestro BPMN skips its real execution and injects your values downstream instead.

Nodes you can simulate

Simulations are useful on the following node types:

• Human Task — returns an approved or rejected response without waiting for a real approver.
• HTTP Request — returns a specific API response, including error responses, so you can test your error handling paths.
• Agent — returns the output of an AI agent, so you can test how your workflow handles different agent responses.
• Connection — returns the output of a connection node, so you can test the workflow without making changes in external applications.

Availability and licensing

• Evaluations are available to users with the license types and Automation Cloud organization licensing plans detailed in Licensing.
• Node simulations, mock generation using LLM features inside evaluations, and LLM-judge evaluators are free of charge during preview. Maestro BPMN executions in design consume from the monthly user limits.

已知限制

• Node-level evaluations are not available.
• Maestro BPMN evaluations have no CLI skills support.

相关页面

• 调试
• 模拟
• 在 Maestro 中使用智能体
• 测试 Maestro 流程