Set up automated data refreshes

Introduction

If you want to do frequent or interval data loads for UiPath Process Mining you can set up automated data refreshes. This guide describes how to set up automated data refreshes by using a batch script.

When running the script, the folder structure for a dataset using the dataset name you specified is generated first. Then all tables in the connector are reloaded and cached for the specified module. These cached tables are used to export the output tables from the connector. The output dataset is used to generate a cache for the application on the specified environment.

Prerequisites

You need to have access to the server or Virtual Machine that contains the UiPath Process Mining installation.

You also need to have a connector in a workspace.

Note: The script will impersonate your workspace. You should create a workspace for the connector and make sure that the workspace is kept up to date. The data refresh will use the specified workspace as is, using all uncommitted changes, if any. Therefore it is recommended to have a dedicated workspace for just the data refresh.

Important:

This batch script requires that the build of the workspace and the Superadmin are identical. If you update the build of the Superadmin without changing it for the workspace (or vice versa), the script will fail, resulting in no refreshed data.

Therefore it is recommended to update the build at all places (Superadmin, connector and other applications) at the same time.

Step 1: Preparation

Create a batch script that you want to use for the automated data refreshes from the code sample below and save it in the root folder of your UiPath Process Mining installation.

Note: It is recommended to include the name of the process and the name of the system in the batch file name.

set MAINDIR=%~dp0
set DATASET=FullData
set ENVIRONMENT=production
set WORKSPACE=Peter/Connector
mkdir "%MAINDIR%\datax\_datasets\%DATASET%"
mkdir "%MAINDIR%\datax\_datasets\%DATASET%\_org"
echo. > "%MAINDIR%\datax\_datasets\%DATASET%\nomodcodes.marker"
for %%M in (..) do (
              call "%MAINDIR%\builds\processgold.bat" -dataserver -workspace "%WORKSPACE%" "" -CCDB app="%%M" o= refreshmvncaches=true
              mkdir "%MAINDIR%\datax\_datasets\%DATASET%\_org\MVNCache" 
              call "%MAINDIR%\builds\processgold.bat" -dataserver -workspace "%WORKSPACE%" "" -export app="%%M" out="%MAINDIR%\datax\_datasets\%DATASET%\_org\MVNCache" o= refreshmvncaches=false reusecache=true onlymarked=true
)
call "%MAINDIR%\builds\processgold.bat" -dataserver -skipmail -env "%ENVIRONMENT%" -CCDB o=* dataset="%DATASET%" p=*

Step 2: Set the Parameters in the Script

Below is a description of the parameters you need to specify in the script.

Parameter	Description
DATASET	the name of the output dataset you want to generate from the script. This dataset will show up in the Superadmin Development data tab.
ENVIRONMENT	the environment for which you want to generate a cache. Note: Usually this will be the environment on which the AppOne for your process is active.
WORKSPACE	the path to the workspace where the connector is located. This is your Login name followed by the name of the Workspace. If the script cannot find the correct workspace, try using forward slash in stead of a backslash.

WORKSPACE Name

Follow these steps to determine the path to the workspace.

Step	Action
1	Go to the UiPath Process Mining installation folder.
2	Go to the workspacesfolder.
3	Go to the [your Login name] folder. The folder contains a folder for each of your workspaces.

Important:

When your Login name contains special characters, the username folder is encoded for HTML. However, for use in the script the % needs to be escaped using three additional %s.

For example: if your username is [e-mail] this will turn into the folder test%2Eexample%40uipath%2Ecom. In the script the username should be specified as test%%%%2Eexample%%%%40uipath%%%%2Ecom. If the username contains an underscore ('_'), this underscore needs to be escaped with %%%%5F.

Step 3: Add the Connector Module Code

Some connectors have multiple modules, therefor you need to specify the module code for the connector for which you want to implement automated data refreshes.

Add the module code of your connector in the for %M in (…) do loop.

Follow these steps to find the module code and add it to the script.

Step	Action
1	Open the connector in your development environment.
2	Click on the logo icon and select Modules....
3	Copy the module code from the Application Code field.
4	Click on CANCEL to close the Modules dialog.
5	Close the connector.
6	In the script replace the "..." inside the ( ) with the module code of your connector in the for %M in (…) do loop.

Step 4: Schedule a Task for the Synchronization

You can use Windows Task Scheduler to schedule a task that runs the batch script for automated data refreshes.

Note: It is recommended to select the local SYSTEM user account for running the task and to select the Run with highest priority option in the Create Task window when creating the task in Windows Scheduler.

The time taken for this task will depend highly on the data volumes moving through your connector. If the task is running for too much time, you are advised to consider the following things, in order:

• Minimize the data volume going into the connector;
• Check for erroneous join conditions which cause data duplication;

• Remove any unused attributes in the connector (every additional attribute in the connector increases the runtime of the script).

  Important: It is strongly advised to cancel the task after it has been running for more than 3 days.

Format of the Dataset

The datasets for use with Process Mining have a specific format, that depends on a marker. This marker is stored at the root of the dataset. See the illustration below for an example.

The marker is just an empty file and is only used for its name. There are two different markers: nomodcodes.marker and usemodcodes.marker.

The usemodcodes.marker is only applicable when data sharding is used to scope the data.

Nomodcodes.marker

The nomodcodes.marker uses the data structure as displayed below.

<DATASET NAME>\<ORGANIZATION CODE>\MVNCache

Below is a description of the elements of the nomodcodes.marker.

Element	Desctiption
<DATASET NAME>	The name that is displayed in the Superadmin Development data tab.
<ORGANIZATION CODE>	The default value is: _org , which will be for 99% of the cases

Usemodcodes.marker

The usemodcodes.marker uses the data structure as displayed below.

<DATASET NAME>\<ORGANIZATION CODE>\<MODULE CODE>\MVNCache

Below is a description of the elements of the usemodcodes.marker.

Element	Desctiption
<DATASET NAME>	The name that is displayed in the Superadmin Development data tab.
<ORGANIZATION CODE>	The default value is: _org , which will be for 99% of the cases
<MODULE CODE>	The code that set in the application. See Modules. This option enables you to use multiple application codes in one app.

The script provided in Step 1. applies to nomodcodes.marker. For usemodcodes.marker you should use the script below.

set MAINDIR=%~dp0
set DATASET=FullData
set ENVIRONMENT=production
set WORKSPACE=Peter/Connector
mkdir "%MAINDIR%\datax\_datasets\%DATASET%"
mkdir "%MAINDIR%\datax\_datasets\%DATASET%\_org"
echo. > "%MAINDIR%\datax\_datasets\%DATASET%\usemodcodes.marker"
for %%M in (..) do (
              call "%MAINDIR%\builds\processgold.bat" -dataserver -workspace "%WORKSPACE%" "" -CCDB app="%%M" o= refreshmvncaches=true
              mkdir "%MAINDIR%\datax\_datasets\%DATASET%\_org\%%M\MVNCache" 
              call "%MAINDIR%\builds\processgold.bat" -dataserver -workspace "%WORKSPACE%" "" -export app="%%M" out="%MAINDIR%\datax\_datasets\%DATASET%\_org\%%M\MVNCache" o= refreshmvncaches=false reusecache=true onlymarked=true
)
call "%MAINDIR%\builds\processgold.bat" -dataserver -skipmail -env "%ENVIRONMENT%" -CCDB o=* dataset="%DATASET%" p=*