Queue Item Retention Policy

Overview

Processing queues items generates large amounts of transactions, which may crowd your Orchestrator database rapidly. A retention policy helps you free up the database in an organized manner.

What is a retention policy? It is an agreement to ensure built-in data off-loading capabilities, by setting an action to remove data from your database after a period of time. What to expect? Due to a lighter database, your cloud Orchestrator performs better.

Queue Item Conditions

For the specified queue, the retention policy you configure applies to all queue items that simultaneously meet the following conditions:

• For completed queue items:
  • They have a final status: Failed, Successful, Abandoned, Retried, Deleted.
  • They have not been modified in over X days, X being the retention duration.
• For uncompleted queue items:
  • They have the New status.
  • They have not been modified in over X days, X being the retention duration.

Postponed queue items

Queue items that were in progress and postponed have their retention duration computed as follows: number of days until the defer date + number of days configured for the retention duration.

Example

• You set a deletion action with a 30-day retention duration for Queue A.
• Queue item B belonging to Queue A is postponed by 10 days.

Queue item B will be deleted in 40 days.

Suspended jobs

Custom retention policies are not applied to queue items that are linked to a suspended job until that job is moved to a completed state or until it is deleted or archived.

Example

• You set a deletion action with a 30-day retention duration for Queue A.
• Queue A is linked to Job B, which is suspended.
• Job B is reenabled and completed within 10 days of setting the retention duration for Queue A.

Queue items belonging to Queue A will be deleted in 40 days.

Determining When a Queue Item Was Last Modified

The queue item validation algorithm searches through all queues and determines which queue items meet the conditions based on four properties, in the following order:

• 1 - LastModificationTime
• 2 - EndProcessingTime
• 3 - StartProcessingTime
• 4 - CreationTime

If a queue item does not have a LastModificationTime value (1), or if the value is null, the algorithm looks at the EndProcessingTime value (2). If the EndProcessingTime value is null, the algorithm looks at the StartProcessingTime value (3). If the StartProcessingTime value is null, the algorithm looks at the CreationTime value (4), and applies the policy based on the first non-null value found.

Determining When a Queue Item Is Deleted

The retention is calculated based on calendar days. Therefore, qualified queue items are deleted on the X+1 calendar day, X being the retention duration, and +1 representing the deletion job execution on the following calendar day.

Note that the job may execute at the very beginning of the following calendar day, hence a couple of hours apart from the moment the retention duration ends.

For example, say you set a retention duration of 1 day:

If the last modification date for a queue item is either 10-06-2022 00:01:00 (the first minute in the calendar day) or 10-06-2022 23:59:00 (the last minute in the calendar day), it qualifies for the deletion job that runs on June 12th (June 10th + 1-day retention duration + 1 day after = June 12th).

Therefore:

• we ensure your queue item data is kept for at least 1 calendar day (the retention duration) by archiving it on the next calendar day,
• we aim at ensuring your items are archived by the end of the next calendar day.

Policy Types

These are the types of retention policy:

• The default policy for newly created queues - all transactions that are part of new queues are deleted after 30 days, without the possibility to undo their deletion. This is the built-in option.
• The custom policy - all transactions are deleted or archived after a retention duration of your choosing. This option can be configured as instructed in the Configuring a custom retention policy section.
  • Completed queue items:
    • The default duration is 30 days.
    • The maximum duration is 180 days.
  • Uncompleted queue items:
    • The default duration is 180 days.
    • The maximum duration is 540 days.

Important: The default retention policies apply to all queues in the tenant.

Policy Outcomes

A custom retention policy has the following outcomes:

• It deletes the valid queue items that are older than the specified duration.
• It deletes the valid queue items that are older than the specified duration, but archives their data into an existing storage bucket, for future reference. This way, you offload your Orchestrator database without losing the information.
• It preserves the unique references of queue items, to guarantee the occurrence of validations after the policy applied.
  Note: Insights dashboards containing deleted queue item information will continue to display the correct data.

Queues Page

The retention policy is not reflected on the Queues page right away.

Data recalculation is triggered by any queue item event, such as Create, Edit, Delete, Status, or Retry. Deleted queue items are no longer included in the recalculated listing.

Configuring a Custom Retention Policy

To configure a custom retention policy:

1. In Orchestrator, navigate to the desired folder in your tenant.
2. Open the Queues page.
3. To add a new queue, click Add Queue. Respectively, to edit an existing queue, click More Actions > Edit for the desired queue. The Create/Update Queue page opens.
4. In the Retention policy section, select the outcome of your policy from the Action dropdown menu.

   You can set different policy outcomes and retention durations for completed and uncompleted queue items.

   To delete queue items, but keep their information, read the steps in the Archiving queue items section.

   To permanently delete queue items, read the steps in the Deleting queue items section.

Archiving Queue Items

If you do not want to lose your queue items data, but you need to offload this information from the Orchestrator database, archive your queue items.

Prerequisite: You need a storage bucket to store your archived queue items.

1. Select Archive from the Action dropdown menu.
2. Select a Retention duration for both completed and uncompleted queue items.

   For completed queue items, you can choose a value between 1 and 180, with 30 being the default.

   For uncompleted queue items, you can choose a value between 180 and 540, with 180 being the default.

   At the end of this duration, all queue items (including queue item events and comments) that have not been updated in the meantime are deleted, and their information is stored in the target bucket.

3. Select a Target bucket to store your archived items.

To retrieve the archived information, access the archive files from the associated storage bucket.

Note:

Note 1: You can either use an Orchestrator storage bucket, or link an external storage bucket.

Note 2: The storage bucket you use must not be read-only, so that the archiving operation can add items to it.

Note 3: You can use the same storage bucket to archive queue items from different queues.

Note 4: This field is only available for the Archive option.

Note 5: Specific and Output Data of encrypted queue items is visible in the storage bucket, as the archiving operation decrypts data upon retrieval and exports it to the target storage.

Note 6: A successful archiving operation is logged on the Tenant > Audit page, identifiable by the Action type as Archive.

Note 7: If an error interrupts the archiving operation, an alert informs you in order to fix the error. The archiving operation is retried the next time the deletion job runs (the next calendar day). Until the archiving is successfully retried, the affected queue items cannot be viewed or accessed.

Archive Output

The .zip file

When you archive your queue items, a .zip file is created at the end of the retention duration with the path:

"Archive/Queues/Queue-{queue_key}/{archiving_operation_date}-{archiving_operation_timestamp}.zip", in which:

• {queue_key} - the unique identifier of the queue containing the queue items
• {archiving_operation_date} - the UTC date when the archive was generated, in the yyyy-MM-dd format
• {archiving_operation_timestamp} - the UTC time when the archive was generated, in the HH-mm-ss-fff format
  For example, an archive file could be named Archive/Queues/Queue-1d1ad84a-a06c-437e-974d-696ae66e47c2/2022-05-26-03-00-08-496.zip.

The .csv file

Once extracted, the .zip file displays a .csv file with the same name syntax:

"Queue-{queue_key}-{archiving_operation_date}-{archiving_operation_timestamp}.csv".

The .csv file contains the following information about your archived queue items:

The Metadata.json file

The .json file contains details about the container queue, to help you identify it more easily.

Large Data Volumes

For queues that processed a large number of transactions, queue items are archived in batches. In this case, the .zip file of each batch has a different {archiving-operation-timestamp}, depending on the time the batch archive was created.

Deleting Queue Items

If you decide that processed queue items data is no longer useful, you can remove all that information from your Orchestrator database.

1. Select Delete from the Action dropdown menu.
2. Select a Retention duration for both completed and uncompleted queue items.

   For completed queue items, you can choose a value between 1 and 180, with 30 being the default.

   For uncompleted queue items, you can choose a value between 180 and 540, with 180 being the default.

   At the end of this duration, all queue items (including queue item events and comments) that have not been updated in the meantime are permanently deleted.

Queue retention policy APIs

To incorporate the retention policy in your client, use the dedicated endpoints of the QueueRetention API in your Swagger file:

• GET /odata/QueueRetention - returns the list of all active policies, containing information such as the policy action, the retention duration in days, the ID of the queue the policy applies to.
• GET /odata/QueueRetention({key}) - returns the policy information about the specified queue.
• PUT /odata/QueueRetention({key}) - updates the policy information about the specified queue.
• DELETE /odata/QueueRetention({key}) - resets the specified queue policy to the default one of 30-day retention + deletion.

Note: If you call the DELETE endpoint for queues created before the introduction of the retention policy feature, the built-in retention policy of 30 days + deletion applies. See an example in our reference guide.

Policy Tracking Columns and Audit

To easily identify which queues have a custom retention policy in place, enable the Retention action and Retention (days) columns on the Queues page, by selecting the corresponding checkboxes from the Columns dropdown.

The Retention action column displays the policy outcome, while the Retention (days) column display the remaining time until the policy applies.

As mentioned, a 30-day retention policy applies to newly created queues. However, you cannot always rely on this value to identify the queues which have a default policy in place. For example, if you set a custom retention duration of 55 days and you later update it to 30 days, the resulting policy is not the default one. To see whether these scenarios represent default policies or not, check the Audit page.

1 represents the Archive action type

0 represents the Delete action type