Subscribe

UiPath Document Understanding

UiPath Document Understanding

Evaluation Pipelines

An Evaluation Pipeline is used to evaluate a trained ML model.

Evaluate a trained model

Configure the evaluation pipeline as follows:

  • In the Pipeline type field, select Evaluation run.
  • In the Choose package major version field, select a major version for your package.
  • In the Choose package minor version field, select a minor version you want to evaluate.
  • In the Choose evaluation dataset field, select a representative evaluation dataset.
  • In the Enter parameters section, there is one environment variable is relevant for Evaluation pipelines you could use:
  • eval.redo_ocr which, if set to true, allows you to rerun OCR when running the pipeline to assess the impact of OCR on extraction accuracy. This assumes an OCR engine was configured when the ML Package was created.
  • The Enable GPU slider is disabled by default, in which case the pipeline is runs on CPU. We strongly recommend that Evaluation pipelines run only on CPU.
  • Select one of the options when the pipeline should run: Run now, Time based or Recurring.
700
  • After you configure all the fields, click Create. The pipeline is created.

Artifacts

For an Evaluation Pipeline, the Outputs pane also includes an artifacts / eval_metrics folder which contains two files:

1479
  • evaluation_default.xlsx is an Excel spreadsheet with three different sheets:
  • The first sheet presents a summary of the overall scores and the scores per batch, for each field, Regular, Column, and Classification fields. A percentage of the perfectly extracted documents is also provided for both per batch and overall documents.
Field NameDescription
FieldName_confidenceProvides the ML model extraction confidence. This is a subjective number, but it can be useful as a fall back for detecting errors in case no business rules are available for the given field.
FieldName_scoreAccuracy of the prediction as compared to the labelled value:
- cell background is white = prediction is exactly correct, accuracy score is equal to 1.
- cell background is yellow = prediction is partially correct, accuracy score is between 0.5 and 1.
- cell background is red = prediction is incorrect, accuracy score is between 0 and 0.5.
In general, good predictions tend to have high confidence, and bad predictions tend to have lower confidence, but this is not guaranteed. It is quite frequent that good predictions will have low confidence. The reverse is less likely, but still possible.
  • The second sheet presents a side by side, color coded comparison of Regular Fields, for increasing document accuracy. The most inaccurate documents are presented at the top to facilitate diagnosis and troubleshooting.
  • The third sheet presents a side by side color, coded comparison of the Column Fields.
  • All scores presented in the Excel file represent accuracy scores.
  • evaluation_metrics_default.txt contains the F1 scores of the predicted fields.

Updated a day ago


Evaluation Pipelines


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.