Managing pipelines

Creating pipelines

1. In the Pipelines page, click the Create new button. The Create new pipeline run page is displayed.

   Alternatively, in the ML Package Details of a specific package, select the Pipeline Runs, and then click the Create new button. The Create new pipeline run page is displayed.

2. In the Create new pipeline run page, select the type of the pipeline run. The available options are Training run, Evaluation run, and Full pipeline run.
3. Add a display name for the pipeline run.
4. Select the package for the pipeline run.
5. Select the package's major and minor versions.
6. Select datasets. According to the selected pipeline type, the following datasets must be specified:
   • For training pipeline, specify the input dataset.
   • For evaluation pipeline, specify the evaluation dataset.
   • For full pipeline run, specify the input dataset and the evaluation dataset.
7. Optional: Enter the parameters for the pipeline runs. Click Add new to display the parameters section, then enter the environment variable and its corresponding value. Multiple parameters are accepted.
8. Select whether the pipeline requires a GPU. By default it is set to No.
   Note: If you are using the AI Units licensing model, the hourly cost of AI Units is displayed under the Enable GPU toggle button. Depending on whether you choose to use GPU or not, the hourly cost changes.
   Note: You can queue pipelines based on the GPU resources. For more information, see Configuring Queuing for GPU resources.
9. Select when the pipeline should run. The possible options are:
   • Run now - the pipeline starts running immediately after its creation.
   • Time based - the pipeline starts running at the date and time you specify in the Date and Time fields.
   • Recurring - the pipeline starts running according to the recurring schedule you set up in the Set Recurring Schedule window. You can set it up to run on specific weekdays at a certain time, or you can use cron expressions.

   Read here more information about scheduling pipelines.

10. Click Create to create the pipeline or Cancel to abort the process. The Create new pipeline run page is closed.
    
    

    

    
    

The pipeline is created and displayed along with its details in the Pipelines page and in the selected package's ML Package Details page's Pipeline Runs tab. The pipeline runs according to the timeframe you chose while creating it.

Scheduling pipelines

During creation, any pipeline type can be either scheduled at a single future date and time, or with a recurring schedule. For example, a schedule may be set for a pipeline to execute on Sunday nights at 1am. This allows for models to be continuously updated as data is sent back from Human-In-the-Loop tasks and additionally allows for more efficient usage of AI Units licenses.

⏲ To create a pipeline to be scheduled at a single future date, select the Time based option in the Create new pipeline run page.

📅 To create a pipeline with a recurring schedule, select the Recurring option in the Create new pipeline run page.

⚙ A more complex schedule than daily at some time can be set by selecting the Advanced tab in the Set Recurring Schedule window and entering a cron expression. There are many free online tools to easily generate cron expressions.

Configuring Queuing for GPU resources

There are resource-limited clusters, especially considering the number of deployed GPUs.

The queuing mechanism manages GPU usage requests, monitors GPU status, and executes the requests when a GPU becomes available. Resource management improves efficiency on tracking and re-initiating requests.

To configure Queuing for GPU resources:

1. Go to ArgoCD AICenter Applications.
   
   

   

   
   
2. Click the App Details button to see the application details.
   
   

   

   
   
3. Click the Parameters tab to see the parameters and their details.
   
   

   

   
   

• global.waitQueue.gpuCount Set to the number of GPU resources available for model training.
• global.waitQueue.queueLength Set to the maximum queue length. Beyond it, new requests will not be queued. The recommended queue length is 5*numGPUs.

Feature enabling

Set the global.waitQueue.enabled property to true.

Configuration

Optional: Change global.waitQueue.gpuCount or global.waitQueue.queueLength.

Functionality

The feature is now enabled. If you trigger 2 consecutive GPU pipelines, one is waiting for the other to complete.

Editing scheduled pipelines

You can only edit pipelines that haven't run yet.

1. Go to the Pipeline Details page.
   • In the Pipelines page, click ⁝ next to a scheduled pipeline and select Details.
   • In the ML Package Details of a specific package, select the Pipeline Runs tab, click ⁝ next to a scheduled pipeline and select Details. The Pipeline Details page is displayed.
2. Click Edit pipeline. The Edit pipeline run page is displayed.
3. You can change the name of the pipeline, the data directory of the scheduled pipeline, as well as the recurring schedule.
4. Click Submit to save the changes. The scheduled pipeline is displayed in the Pipelines page with its updated information.

Removing pipelines

1. Remove a pipeline.
   • In the Pipelines page, click ⁝ next to a pipeline and select Remove.
   • In the ML Package Details of a specific package, select the Pipeline Runs tab, click ⁝ next to a pipeline and select Remove. A confirmation dialog is displayed.
2. Click OK to delete the pipeline.
   Note:

   Removing a Packaging, Waiting for resources or Running pipeline first stops it and then removes it.

   If you just want to stop a Packaging, Waiting for resources or Running pipeline and look at its logs, click ⁝ next to it and select the Details option to navigate its corresponding Pipeline Details page. You can stop the pipeline from there.

Viewing pipeline details

You can consult more information about a specific pipeline or perform other actions.

• In the Pipelines page, click ⁝ next to a scheduled pipeline and select Details.

• In the ML Package Details of a specific package, select the Pipeline Runs tab, click ⁝ next to a scheduled pipeline and select Details. The Pipeline Details page is displayed.
• Or, if an ML Package is generated by a pipeline, you can access all information related to the pipeline directly from the ML Package window by clicking on the three dots menu ⁝ and Pipeline Details. This is only available for new pipelines.

The information displayed and the actions you can perform here depend on the pipeline status.

Scheduled pipeline details

The Pipeline Details page for Scheduled pipelines shows an information tab.

You can perform the following actions:

Delete the pipeline.

Edit the pipeline. In the displayed Edit pipeline run page, any fields can be updated. Selecting Run Now does not create a new pipeline, instead, it executes this pipeline immediately, thus removing it from the list of scheduled pipelines.

Packaging pipeline details

The Pipeline Details page for Packaging pipelines shows an information tab.

You can perform the following action:

Kill the pipeline. This stop the execution of the pipeline, so it won't run and it won't consume AI Units. Its status changes to Killed.

Waiting for resources pipeline details

The Pipeline Details page for Waiting for resources pipelines shows an information tab.

You can perform the following action:

Kill the pipeline. This removes the pipeline from the queue, so it won't be executed and it won't consume AI Units. Its status changes to Killed.

Running pipeline details

The Pipeline Details page for Running pipelines shows an information tab and real-time logs.

You can perform the following action:

Kill the pipeline. This immediately stops the pipeline and changes its status to Killed. The logs show the snapshot at the time the kill action was executed.

Failed pipeline details

The Pipeline Details page for Failed pipelines shows an information tab and logs. Depending on the stage at which this pipeline failed, (partial) pipeline outputs are displayed.

You can perform the following action:

Restart the pipeline. This adds a new pipeline to the queue, with the exact same parameters with which it was created. If the queue is empty, the pipeline immediately starts executing.

Killed pipeline details

The Pipeline Details page for Killed pipelines shows an information tab. In addition, depending on the point at which this pipeline was killed, the pipeline details page may also include logs.

You can perform the following actions:

Restart the pipeline.

Remove the pipeline.

Successful pipeline details

The Pipeline Details page for Successful pipelines shows an information tab, logs and pipeline outputs.

You can perform the following actions:

Delete the pipeline.

Remove the pipeline.

Restart the pipeline.

Logs

You can download a report of the pipeline run from the Logs section. To do so, click on the Download Pipeline Report button.

We recommend attaching this report when submitting an issue for faster troubleshooting.

This report gathers all the necessary information required to debug an issue, including account ID, tenant ID, AI Units, and the respective package and pipeline information.

Check the screenshot below for pipeline run report example.