Evaluations (Preview)

Automated quality checks for Maestro BPMN workflows, with evaluation sets, deterministic and LLM-judge evaluators, and node output simulations.

Evaluations let you define automated quality checks for a Maestro BPMN workflow. You group expected outcomes into an evaluation set, and each execution is scored against the evaluators you attach.

Evaluations use the same evaluation framework already available for AI agents and Maestro Flow, so workflows with non-deterministic steps can be verified before they ship.

概念

• Dataset — a collection of data points that target the same subject. Running the set scores the workflow against every data point at once.
• Data point — a single case: a set of input values and the expected output for those inputs. You can author a data point by hand, or capture one from an actual run.
• Evaluator — a grader that compares actual output against expected output, or the trajectory against an expected trajectory, and produces a score.
• スコア — 評価器がデータ ポイントについて報告する結果です。

BPMN-level evaluation sets

An evaluation set targets the whole Maestro BPMN workflow. It is pinned to a trigger entry point, and each data point supplies the trigger's inputs and the workflow's expected output.

Node-level evaluation sets pin an evaluation to a single node, so that each data point supplies that node's inputs and its expected output. Node-level evaluations are not part of this public preview.

評価器

Maestro BPMN evaluators fall into two families.

決定論的

These evaluators compare output to expected output exactly, with no model in the loop.

• 次の値を含む — 出力に予期される部分文字列が含まれていることを確認します。
• 完全一致 — 出力が期待値と等しいことを確認します。
• JSON の類似性 — 出力と期待される出力を JSON として比較します。

LLM による評価

These evaluators use a model to judge the output against the expected output.

• 出力の意味的類似性 — 出力が期待される出力と同じ意味を持つかどうかを判断します。
• 出力: 厳密な JSON の類似性 — JSON 出力を期待される JSON に照らして判断します。
• 軌跡の類似性 — エージェントが実行したステップを、予期される軌跡に照らして判断します。
• 軌跡シミュレーション — シミュレートされた軌跡を判断します。

実行からデータ ポイントをキャプチャする

You don't have to author every data point by hand. You can build one by importing an actual execution: when you capture a debug run, its inputs and outputs become the data point's input and expected output.

Capturing a run is a fast way to seed an evaluation set from real behavior you have already observed.

When to use evaluations

評価は次の場合に最も役に立ちます。

• You are building Maestro BPMN workflows that invoke AI agents and need to verify that the agent's output meets quality expectations before publishing.
• You need regression coverage, and running evaluations after each change confirms that existing behavior has not been broken.
• 同じワークフローの 2 つのバージョンで評価を実行して、代替の実装を比較する場合を考えます。

評価結果

Results are stored in the evaluation history and can be reviewed at any time. They are visible to the users who run them, and can be exported as either JSON or CSV. Individual traces inside an evaluation run can also be exported.

The evaluation runs history tab shows, for each run:

• Per-evaluator scores, together with the grader's justification
• Individual data point results, with actual and expected values side by side
• The full trace: what triggered the run, the path it took, per-node output, and timing

The trace and the grader justification indicate what needs to change for the workflow to run correctly on each case.

Example: classifying support emails

A Maestro BPMN workflow uses an AI agent to classify customer support emails. Before publishing, you set up an evaluation set with 10 data points:

• 5 emails classified as "billing"
• 3 emails classified as "technical"
• 2 ambiguous emails, treated as edge cases

Each data point defines the expected classification output value, or a triggered escalation to a human for the ambiguous emails. An Exact match evaluator on the classification scores each result for "billing" or "technical", and a trajectory evaluator checks the human escalation. Each run shows which cases the agent got wrong before the workflow goes to production.

Node simulations

Simulations override selected node outputs during an evaluation run, so you can test downstream logic without executing a node's real action. This is useful when the real action is slow, expensive, or depends on external systems that are not available during design.

When a simulation is enabled on a node, the run skips that node's actual execution and uses the mocked output values directly. All downstream nodes receive the mocked values as if the node had run normally.

Simulations apply only during test runs. They are never active in published, production workflows.

Simulation strategies

Each simulated node uses one of two strategies:

• Static mock — a rule-based simulation that always returns the value you define.
• Generated mock — an LLM-generated response, produced dynamically each time the evaluation runs.

Setting up a simulation

Configure a simulation on any node whose real output you want to replace during an evaluation run. The following sequence shows the full flow, from selecting the node through entering the simulated output.

1. Select the node you want to simulate on the canvas.
2. In the configuration panel, open Add simulation.
3. Choose the simulation strategy for the node.
4. Enter the simulated output for the strategy you chose:
   1. For Static mock, enter the output values, either by editing the JSON manually or by generating it with the prompt feature. Use valid JSON that matches the node's output schema.
   2. For Generated mock, enter the prompt the model uses to generate a tool response on each evaluation run.

Result: The node is simulated. On the next test run, Maestro BPMN skips its real execution and injects your values downstream instead.

Nodes you can simulate

Simulations are useful on the following node types:

• Human Task — returns an approved or rejected response without waiting for a real approver.
• HTTP Request — returns a specific API response, including error responses, so you can test your error handling paths.
• Agent — returns the output of an AI agent, so you can test how your workflow handles different agent responses.
• Connection — returns the output of a connection node, so you can test the workflow without making changes in external applications.

Availability and licensing

• Evaluations are available to users with the license types and Automation Cloud organization licensing plans detailed in Licensing.
• Node simulations, mock generation using LLM features inside evaluations, and LLM-judge evaluators are free of charge during preview. Maestro BPMN executions in design consume from the monthly user limits.

既知の制限事項

• Node-level evaluations are not available.
• Maestro BPMN evaluations have no CLI skills support.

関連ページ

• デバッグ
• シミュレートする
• Maestro でエージェントを使用する
• Maestro プロセスをテストする