Training Pipelines

Important:

Minimal dataset size

For successfully running a Training pipeline, we strongly recommend at least 25 documents and at least 10 samples from each labeled field in your dataset. Otherwise, the pipeline throws the following error: Dataset Creation Failed.

Training on GPU vs CPU

• For larger datasets, you need to train using GPU. Moreover, using a GPU (AI Robot Pro) for training is at least 10 times faster than using a CPU (AI Robot).
• Training on CPU is only supported for datasets up to 5000 pages in size for ML Packages v21.10.x and up to 1000 pages for other versions of ML Packages.
• CPU training was limited to 500 pages before 2021.10, it went up to 5000 pages for 2021.10 and with 2022.4 it will come back down to 1000 pages max.

There are two ways to train an ML model:

• training a model from scratch
• retraining an out-of-the-box model

Training a model from scratch can be done using the DocumentUnderstanding ML package, which does this on the dataset provided as input.

Retraining can be done using out-of-the-box ML packages such as Invoices, Receipts, Purchase Orders, Utility Bills, Invoices India, Invoices Australia etc., basically, any other data extraction ML Package except for DocumentUnderstanding. Training using one of these packages has one additional input: a base model. We refer to this as retraining because you are not starting from scratch but from a base model. This approach uses a technique called Transfer Learning where the model takes advantage of the information encoded in another model, the preexisting one. The model retains some of the out-of-the-box knowledge, but it learns from the new data too. However, as your training dataset size increases, the pretrained base model matters less and less. It is mainly relevant for small to medium size training datasets (up to 500-800 pages).

Configure the training pipeline as follows:

• In the Pipeline type field, select Train run.
• In the Choose package field, select the package you created based on the DocumentUnderstanding ML Package.
• In the Choose package major version field, select a major version for your package.
• In the Choose package minor version field, select a minor version for your package. It is strongly recommended to always use minor version 0 (zero).
• In the Choose input dataset field, select a representative training dataset.
• In the Enter parameters section, enter any environment variables defined, and used by your pipeline, if any. For most use cases, no parameter needs to be specified; the model is using advanced techniques to find a performant configuration. However, here are some environment variables you could use:
• auto_retraining which allows you to complete the Auto-retraining Loop; if the variable is set to True, then the input dataset needs to be the export folder associated with the labeling session where the data is tagged; if the variable remains set to False, then the input dataset needs to correspond to the dataset format.
• model.epochs which customizes the number of epochs for the Training Pipeline (the default value is 100).
  Note: For larger datasets, containing more than 5000 pages, you can initially perform a full pipeline run with the default number of epochs. This allows you to evaluate the model’s accuracy. After that, you can decrease the number of epochs to about 30-40. This approach allows you to compare the accuracy of the results and determine if the reduction of epochs yields comparable precision.

  When using smaller datasets, in particular those with fewer than 5000 pages, you can maintain the default number of epochs.

• Select whether to train the pipeline on GPU or on CPU. The Enable GPU slider is disabled by default, in which case the pipeline is trained on CPU.
• Select one of the options when the pipeline should run: Run now, Time based or Recurring. In case you are using the auto_retraining variable, select Recurring.
  
  

  

  
  
• After you configure all the fields, click Create. The pipeline is created.