Running an evaluation

Run an evaluation of the IXP model in the Extract node of UiPath Maestro™ Flow on the annotated documents of the dataset, and track the changes that are not evaluated yet.

Prerequisites

• Annotated documents in the dataset of the node. You need ground truth for a representative set of documents to compare the predictions of the model against.

Running an evaluation

To run an evaluation, proceed as follows:

1. Hover over the Flow Extract node, and select Open full view.

2. In the Dataset tab or the Eval runs tab, select Run evaluation. The Configure evaluation run pane opens.

   

   Alternatively, select Run evaluation in the Definition section of the properties pane of the node.

3. In the Configure evaluation run pane, check the following options:

   • Dataset - The dataset that the run evaluates. Evaluation target shows the node that the run evaluates.
   • Filter by tags - Optionally, select tags to evaluate only a subset of the documents in the dataset. By default, the run evaluates All documents.
   • Evaluators - The evaluators that score the run.

4. If the pane shows the Review these data points before running the evaluation warning, review the listed documents. For example, the warning lists documents whose expected output does not match the current output schema.

5. Optionally, expand Run settings, and configure the following options:

   • Name - The name of the evaluation run. By default, the name is the name of the dataset, followed by the current date.
   • Batch size - A value from 1 to 50. The default value is 10.
   • Timeout - A value from 5 to 500 minutes. The default value is 20 minutes.

   

6. Select Save & run.

To check the results of the run, navigate to the Eval runs tab. For details, check Evaluating model performance.

Evaluation runs on a subset of documents do not affect the health score of the node. For details, check Node health score.

Changes not evaluated

The Changes not evaluated label appears in the Flow Extract node when changes are made to the node, such as a model configuration change, a taxonomy change, or an annotation change. These changes affect the input that the extraction model receives, so the label alerts you that there are changes not yet evaluated.

To view the changes, hover over the Changes not evaluated label. The list shows the number of changes and each change, for example, Model configuration changed. To evaluate the changes, select Run evaluation.