Evaluations (Preview)

Automated quality checks for Maestro BPMN workflows, with evaluation sets, deterministic and LLM-judge evaluators, and node output simulations.

Evaluations let you define automated quality checks for a Maestro BPMN workflow. You group expected outcomes into an evaluation set, and each execution is scored against the evaluators you attach.

Evaluations use the same evaluation framework already available for AI agents and Maestro Flow, so workflows with non-deterministic steps can be verified before they ship.

Konzepte

• Dataset — a collection of data points that target the same subject. Running the set scores the workflow against every data point at once.
• Data point — a single case: a set of input values and the expected output for those inputs. You can author a data point by hand, or capture one from an actual run.
• Evaluator — a grader that compares actual output against expected output, or the trajectory against an expected trajectory, and produces a score.
• Punktzahl – das Ergebnis, das ein Auswerter für einen Datenpunkt meldet.

BPMN-level evaluation sets

An evaluation set targets the whole Maestro BPMN workflow. It is pinned to a trigger entry point, and each data point supplies the trigger's inputs and the workflow's expected output.

Node-level evaluation sets pin an evaluation to a single node, so that each data point supplies that node's inputs and its expected output. Node-level evaluations are not part of this public preview.

Auswerter

Maestro BPMN evaluators fall into two families.

Deterministisch

These evaluators compare output to expected output exactly, with no model in the loop.

• Enthält – überprüft, ob die Ausgabe eine erwartete Teilzeichenfolge enthält.
• Exakte Übereinstimmung – überprüft, ob die Ausgabe dem erwarteten Wert entspricht.
• JSON-Ähnlichkeit – vergleicht die Ausgabe und die erwartete Ausgabe als JSON.

LLM-Judge

These evaluators use a model to judge the output against the expected output.

• Semantische Ähnlichkeit der Ausgabe – beurteilt, ob die Ausgabe dasselbe bedeutet wie die erwartete Ausgabe.
• Ausgabe der strengen JSON-Ähnlichkeit – bewertet die JSON-Ausgabe anhand der erwarteten JSON.
• Trajektorien-Ähnlichkeit – bewertet die Schritte, die der Agent ausgeführt hat, anhand einer erwarteten Trajektorie.
• Trajektorien-Simulation – bewertet einen simulierten Verlauf.

Erfassen eines Datenpunkts aus einer Ausführung

You don't have to author every data point by hand. You can build one by importing an actual execution: when you capture a debug run, its inputs and outputs become the data point's input and expected output.

Capturing a run is a fast way to seed an evaluation set from real behavior you have already observed.

When to use evaluations

Auswertungen sind in folgenden Fällen am nützlichsten:

• You are building Maestro BPMN workflows that invoke AI agents and need to verify that the agent's output meets quality expectations before publishing.
• You need regression coverage, and running evaluations after each change confirms that existing behavior has not been broken.
• Sie möchten alternative Implementierungen vergleichen, indem Sie Auswertungen an zwei Versionen desselben Workflows ausführen.

Auswertungsergebnisse

Results are stored in the evaluation history and can be reviewed at any time. They are visible to the users who run them, and can be exported as either JSON or CSV. Individual traces inside an evaluation run can also be exported.

The evaluation runs history tab shows, for each run:

• Per-evaluator scores, together with the grader's justification
• Individual data point results, with actual and expected values side by side
• The full trace: what triggered the run, the path it took, per-node output, and timing

The trace and the grader justification indicate what needs to change for the workflow to run correctly on each case.

Example: classifying support emails

A Maestro BPMN workflow uses an AI agent to classify customer support emails. Before publishing, you set up an evaluation set with 10 data points:

• 5 emails classified as "billing"
• 3 emails classified as "technical"
• 2 ambiguous emails, treated as edge cases

Each data point defines the expected classification output value, or a triggered escalation to a human for the ambiguous emails. An Exact match evaluator on the classification scores each result for "billing" or "technical", and a trajectory evaluator checks the human escalation. Each run shows which cases the agent got wrong before the workflow goes to production.

Node simulations

Simulations override selected node outputs during an evaluation run, so you can test downstream logic without executing a node's real action. This is useful when the real action is slow, expensive, or depends on external systems that are not available during design.

When a simulation is enabled on a node, the run skips that node's actual execution and uses the mocked output values directly. All downstream nodes receive the mocked values as if the node had run normally.

Simulations apply only during test runs. They are never active in published, production workflows.

Simulation strategies

Each simulated node uses one of two strategies:

• Static mock — a rule-based simulation that always returns the value you define.
• Generated mock — an LLM-generated response, produced dynamically each time the evaluation runs.

Setting up a simulation

Configure a simulation on any node whose real output you want to replace during an evaluation run. The following sequence shows the full flow, from selecting the node through entering the simulated output.

1. Select the node you want to simulate on the canvas.
2. In the configuration panel, open Add simulation.
3. Choose the simulation strategy for the node.
4. Enter the simulated output for the strategy you chose:
   1. For Static mock, enter the output values, either by editing the JSON manually or by generating it with the prompt feature. Use valid JSON that matches the node's output schema.
   2. For Generated mock, enter the prompt the model uses to generate a tool response on each evaluation run.

Result: The node is simulated. On the next test run, Maestro BPMN skips its real execution and injects your values downstream instead.

Nodes you can simulate

Simulations are useful on the following node types:

• Human Task — returns an approved or rejected response without waiting for a real approver.
• HTTP Request — returns a specific API response, including error responses, so you can test your error handling paths.
• Agent — returns the output of an AI agent, so you can test how your workflow handles different agent responses.
• Connection — returns the output of a connection node, so you can test the workflow without making changes in external applications.

Availability and licensing

• Evaluations are available to users with the license types and Automation Cloud organization licensing plans detailed in Licensing.
• Node simulations, mock generation using LLM features inside evaluations, and LLM-judge evaluators are free of charge during preview. Maestro BPMN executions in design consume from the monthly user limits.

Bekannte Einschränkungen

• Node-level evaluations are not available.
• Maestro BPMN evaluations have no CLI skills support.

Zugehörige Seiten

• Debugging
• Simulieren
• Verwenden von Agents in Maestro
• Maestro-Prozesse testen