Performing Insights database maintenance

Overview

Insights Database works using two tables: dbo and read. Both tables store data from Orchestrator databases and pass data between each other. Over time, tables store a large amount of data that can impact database performance. You can free up space by removing data that is not relevant or outdated.

The estimated threshold for storing historical data is set to

200 million RobotLogs
100 million QueueItemEvents
100 million JobEvents
20 million QueueItems
20 million Job

Use the following database maintenance SQL scripts to create SQL Stored Procedures, then execute these SQL Stored Procedures to either delete all data from [dbo].Jobs, [dbo].JobEvents, [dbo].QueueItems, [dbo].QueueItemEvents, [dbo].RobotLogs tables, or just the [dbo].RobotLogs.

Note: Back up your database before executing the scripts. Before running the

PROCEDURE
                     [read].[Delete_Insights_Data_Read]

and

PROCEDURE
                     [read].[Delete_Process_Logs_Read]

scripts, the Insights module needs to be disabled.

Considerations

Scripts with the _DBO suffix are used to delete data from DBO schema tables,
Scripts with the _Read suffix are used to delete data from Read schema tables.
Large-sized batches might impact the speed of execution. Consider using the default batch size, which is set to 100000.

Prerequisites

Before running the PROCEDURE [read].[Delete_Insights_Data_Read] and PROCEDURE [read].[Delete_Process_Logs_Read] scripts, the Insights module needs to be disabled.

Log in to the VM where Orchestrator is installed.
Find the following: Insights.ModuleEnabled.
Find the following configuration value:
```
"codes": [
    {
      "code": "<add key=\"Insights.ModuleEnabled\" value=\"true\" />",
      "language": "text",
      "name": "configuration value"
    }
  ]
}
  "codes": [
    {
      "code": "<add key=\"Insights.ModuleEnabled\" value=\"true\" />",
      "language": "text",
      "name": "configuration value"
    }
  ]
}
```
Replace true with false.
Save the file.
Go back to the VM where Orchestrator is installed and restart the VM.
Run the maintenance scripts.

Note: Make sure to enable the Insights module back after running the maintenance scripts.

Resources

SQL Stored Procedures download location

SQL stored procedure 1: delete all data based on cut-off timestamp

Generate and save the following SQL Stored Procedures to delete all data types (Jobs, JobEvents, QueueItems, QueueItemEvents, RobotLogs) for a cut-off timestamp.

SQL Stored Procedures	Parameter	Description
Delete_Insights_Data_DBO	@CutoffTimeStamp `DATETIME` (Required) Delete all five types of data before a specified timestamp (e.g., `2021-01-01 01:00:05`). Do not provide a timestamp that deletes all data to an empty table. @BatchSize `INT = 100000` (Optional) Default is set to `100000`. Data is deleted batch by batch. For example, if you have 1 million RobotLogs and leave the default value, the delete command runs ten times to remove all data.	Delete data from `[dbo].[Jobs]`, `[dbo].[JobEvents]`, `[dbo].[QueueItems]`, `[dbo].[QueueItemEvents]`, `[dbo].[RobotLogs]` tables based on a cut-off timestamp.
Delete_Insights_Data_Read	N/A	Truncate `[read].Jobs`, `[read].JobEvents`,`[read].QueueItems`, `[read].QueueItemEvents`, `[read].RobotLogs` tables.

Note:

The Insights Dashboard widgets are available again after SQL Stored Procedure 1 has finished, and the data has been backfilled. Keep in mind that the backfill process may take hours to complete given that you have a large amount of data.

When you first run the Delete_Insights_Data_DBO SQL Script, it creates a QueueItems.IX_CreationTime nonclustered index with the purpose of making future executions faster.

SQL stored procedure 2: delete RobotLogs data based on ProcessNames and Tenant ID

Generate and save this SQL Stored Procedure to delete RobotsLogs based on ProcessNames and Tenant ID.

SQL Stored Procedures Parameter Description

SQL Stored Procedures	Parameter	Description
Delete_Process_Logs_DBO	@ProcessName `NVARCHAR(128)` (Required) The Orchestrator process name. @TenantId `INT` (Required) Tenant ID is required as two tenants might run processes with identical names. @BatchSize `INT = 100000` (Optional) Delete data in batches of 100000. See Considerations.	Delete data from `[dbo].[RobotLogs]` table by ProcessName and Tenant ID.
Delete_Process_Logs_Read	@ProcessName `NVARCHAR(128)` (Required) @TenantId `INT` (Required)	Truncate `[read].RobotLogs` table.

Delete_Process_Logs_DBO

@ProcessName NVARCHAR(128) (Required)

The Orchestrator process name.

@TenantId INT (Required)

Tenant ID is required as two tenants might run processes with identical names.

@BatchSize INT = 100000 (Optional)

Delete data in batches of 100000. See Considerations.

Delete data from [dbo].[RobotLogs] table by ProcessName and Tenant ID.

Delete_Process_Logs_Read

@ProcessName NVARCHAR(128) (Required)
@TenantId INT (Required)

Truncate [read].RobotLogs table.

Note: The Insights Dashboard widgets related to RobotLogs are available again after SQL Stored Procedure 2 has finished, and the data has been backfilled. Keep in mind that the backfill process may take hours to complete given that you have a large amount of data.

Stored Procedures (Database Engine)

On this page