Pipelines

This section provides frequently encountered errors related to Pipelines.

Pipeline Failed due to ML Package Issue

A pipeline run is failing due to ML Package issue.

A potential cause for this error can be a wrong minor version chosen when running the pipeline.

Check the section below for more information on how to choose the correct minor version.

Choosing the Correct Minor Version When Running Pipelines

When deploying a new package, the only minor version available is 0. The reason for this is because there are no pipelines executed on this package yet.

Training pipelines

If you are deploying a training pipeline, we strongly recommend you always use minor version 0.

Full pipelines

If you are deploying a full pipeline, we strongly recommend you always use minor version 0.

Evaluation pipelines

Evaluation pipelines are used to evaluate a trained ML model. You can execute this on any version of the ML Package to get the corresponding evaluation scores. This is similar with grading or evaluating an ML model with the evaluation dataset.

Note: Choosing a minor version depends on the kind of package the evaulation pipeline is being run on. Because of this, special attention is needed when choosing the minor version. Make sure to choose the version that you want to evaluate.

For example, the types of packages that are available are pre-trained out of the box packages and untrained Document Understanding models:

• Pre-trained packages (out of the box packages): since these packages are pre-trained, run the evaluation pipeline on the minor version 0. To evaluate the model after training using specific data, choose the minor version you want to evaluate (trained version).
• Document Understanding:
  • Since these are generic, retrainable models, the models need to be trained first. Only run an evaluation pipeline when the models are trained and a new minor version of the package is available.
  • Select the most recent minor version, or any other minor version (except 0), for which the evaluation scores can be obtained.

Pipeline Killed Automatically

Pipeline is killed automatically

Pipelines are automatically killed after seven days to avoid being stuck for longer periods of time and consuming licenses. Follow the recommendations below.

• Enable GPU.
• Apply dataset optimization technics.
• Reduce the number of EPOCHS.

Pipeline Running For Too Long

If the pipeline is running for too long, you can check the logs. You can encounter the following statuses:

• Waiting for License
• Running
• Failed
• Killed

Check the sections below for more details on each status.

Note: If the logs are streaming, try the following fixes:

• Enable GPU
• Optimize the dataset. For detailed information on Document Understanding datasets, check the Training High Performing Models page from the Document Understanding guide.

Waiting for Licenses

If the pipeline run is stuck in the Waiting for Licenses state, check if the corresponding licenses are available.

1. Open Automation Cloud™.
2. Go to theAdmin > Licenses page.
3. Check if the corresponding AI Units are available.

If AI Units are not available or are consumed, contact our Sales department for procurement. For more information on AI Units, check the AI Units page.

Running

If the pipeline run is stuck in the Running state, follow the steps below.

1. Select the stuck pipeline.
2. Check the Logs section.
   • If the logs are recent and are streaming, the pipeline is in progress.
   • If the last log is generated a long time back, download the logs using the Download button and share it with our Support department. If the download button is not visible or disabled, copy the logs from the Logs section and share it with our Support department.

Failed

If the pipeline run is in the Failed state, check the possible reasons below.

Check that document type data is in dataset folder and follows folder structure

The following error occurs:

Error: Document type data not valid, check that document type data is in dataset folder and follows folder structure.

The format of the folder provided for training needs to be in the dataset format.

1. Make sure that the provided dataset is correct.
2. Make sure that the provided dataset is exported from Document Manager. For more information on datasets related to Document Understanding, check the Export Documents page from the Document Understanding guide.
3. In case of scheduled pipelines for automatic retraining loop, select the folder containing the exports from the Data Labeling sessions and latest.txt.

Images or directory does not exist or is empty for invoices dataset

The pipeline run fails because images/directory does not exist/is empty for invoices dataset.

The dataset path provided for either training dataset or evaulation dataset is empty.

To fix this, update the dataset path for evaluation or training, according to the pipeline.

Unschedulable nodes are available

If you encounter the Unschedulable 0/n nodes are available error, contact our Support department with the your Automation Cloud™ tenant information.

No space left on device

If you encounter the No space left on device error, contact our Support department with the your Automation Cloud™ tenant information.

Killed

The Killed status is usually displayed when the pipeline was killed by the user. For more information on managing pipelines, check the Managing Pipelines page.

If the pipeline status is Killed without any user intervention, the most common reason is that pipelines are automatically killed after seven days. For more information on pipeline statuses, check the About Pipelines page.

Pipelines Failed due to Datasets Issues

A pipeline run is failing due to dataset structure, input parameters, path, folders, or evaluation dataset.

Wrong dataset format

The following error occurs:

#Error: Training and / or test set is empty, verify that training / test split is correctly set in split.csv

This error is most commonly is usually caused by the wrong dataset format or incorrect ratio of train and validation set in split.csv. Check the Training Dataset page for general guidelines on how to create a training dataset.

Evaluation set not provided

The following error occurs:

#Error: Training failed for pipeline type: FULL_TRAINING, error: Full / evaluation pipelines require an evaluation dataset. Please re-run the pipeline providing an evaluation dataset

This error typically occurs when an evaluation dataset was not provided. Check the Training Dataset page for general guidelines on how to create a training dataset.