Évaluations

Evaluations let you define automated quality checks on the outputs of a Flow workflow. You group expected outcomes into an evaluation set, and Flow scores each execution against the evaluators you attach.

Concepts

• Evaluation set — a collection of data points that target the same subject. Run the set to score the workflow against every data point at once.
• Data point — a single case: a set of input values and the expected output for those inputs. Author a data point by hand, or capture one from an actual run (see Capturing a data point from a run).
• Dataset — the data points that make up an evaluation set.
• Evaluator — a check that compares actual output against expected output and produces a score. A boolean evaluator yields pass or fail per evaluator.
• Score — the result an evaluator reports for a data point.

Node-level vs flow-level evaluation sets

An evaluation set targets either a single node or the whole flow:

• Node-level — pinned to one node. Each data point supplies that node's inputs and its expected output.
• Flow-level — pinned to a trigger entry point. Each data point supplies the trigger's inputs and the flow's expected output.

You cannot mix node-level and flow-level data points in the same evaluation set.

Évaluateurs

Flow groups evaluators into three families.

Déterministe

Compare output to expected output exactly, with no model in the loop.

• Contains — checks that the output contains an expected substring.
• Exact match — checks that the output equals the expected value.
• JSON similarity — compares output and expected output as JSON.

LLM-judge

Use a model to judge the output against the expected output.

• Output semantic similarity — judges whether the output means the same as the expected output.
• Output strict JSON similarity — judges JSON output against expected JSON.
• Trajectory similarity — judges the steps the agent took against an expected trajectory.
• Trajectory simulation — judges a simulated trajectory.

Tool-call

Check how a node called its tools. These evaluators are node-level only.

• Argument match — checks the arguments passed to a tool call.
• Call count — checks how many times a tool was called.
• Call order — checks the order in which tools were called.
• Output match — checks a tool call's output.

Evaluation set lifecycle

An evaluation set is built around a subject, data points, and evaluators. The subject is either a single node or a trigger entry point. Each data point supplies input values and the expected output, and each evaluator defines how Flow scores the execution result.

When an evaluation set runs, Flow executes the subject for each data point and displays the per-evaluator scores in the evaluation history.

Capturing a data point from a run

You don't have to author every data point by hand. You can build one by importing an actual execution: capture a debug run, and its inputs and outputs become the data point's input and expected output. This is a fast way to seed an evaluation set from real behavior you've already observed.

Using evaluations in development

Evaluations are most useful when:

• You are building workflows that invoke AI agents and need to verify that the agent's output meets quality expectations before publishing.
• You need regression coverage — running evaluations after each change confirms that existing behavior has not been broken.
• You want to compare alternative implementations by running evaluations on two versions of the same workflow.

Résultats de l’évaluation

Results are stored in the evaluation history and can be reviewed at any time. Each run shows the per-evaluator scores, individual data point results, and the actual vs. expected values for each data point.

Practical example

A workflow that uses an AI agent to classify customer support emails. Before publishing, you set up an evaluation set with 10 data points:

• 5 emails that should be classified as "billing"
• 3 emails that should be classified as "technical"
• 2 edge cases (ambiguous emails)

Each data point defines the expected classification output value. An Exact match evaluator on the classification scores each result, and the evaluation run shows which cases the agent got wrong before the workflow goes to production.

Common mistakes

• Only running an evaluation set once — Evaluation sets are most useful after every significant change to a workflow that invokes an AI agent. Agent behavior can shift as models are updated even if the workflow has not changed.
• Writing data points only for the happy path — Evaluations are most valuable for edge cases and failure modes. Strong data points include inputs that are ambiguous, malformed, or at the boundary of the expected range.
• Treating an all-pass score as a signal to stop testing — Three passing data points is not confidence. Larger sets, especially ones that reflect real inputs from production, provide stronger coverage.

Pages liées

• Output mocking
• Observing runs
• Nœud d'agent
• Orchestrate an AI agent