Loading data using DataUploader

With DataUploader you can upload data files up to 5TB each directly into a Process Mining process app. Loading data using DataUploader is more stable than loading data using the Upload data option in Process Mining, also for smaller data files. If the upload using DataUploader fails, for example, due to an unstable connection, DataUploader will retry the upload up to 4 times.

Note:

Always make sure that the data is in the required format for the app template used for the process app. Refer to App Templates.

Parameters

Below is an overview of the parameters for DataUploader.

Parameter	Format	Description	Mandatory Y/N	Example
-c --csv-dir	<csv dir value>	The directory containing the CSV files you want to upload. Note: This can also be a relative path.	Y	C:\P2P data
-s --sas-url	<sas url value>	The Shared Access Signature (SAS) URL for the Azure Blob Storage container where the files need to be uploaded. See Retrieving the credentials for the Azure blob storage	Y
-e --end-of-upload-api	<value>	URL to the end-of-upload-API that will be called when the files have been successfully uploaded. See Retrieving the credentials for the Azure blob storage	Y
-d --delimiter	<delimiter value>	The ASII code of the delimiter used in the input files. Note: This must be an ascii value between 0 and 127. Default is 9 (Tab).	N	44 (= Comma) 9 (= Tab)
-p --proxy	<proxy value>	The proxy URL	N
-u --proxy-username	<proxy username>	The username if authentication is needed to connect to proxy server.	N
-P --proxy-password	<proxy password>	The password if authentication is needed to connect to proxy server.	N
-r --recursive		Look for all files in the given folder including sub-folders. Attention: If files with duplicate names are present in different folders, the behavior can be unpredictable. Note: By default, only files in the given folder are uploaded and sub-folders are ignored.
-f --config-file	<config file value>	Allows you to use a configuration file instead of inline parameters. Attention: Inline parameters override the values in the config file.
-h		Allows you to display help for a DataUploader command.

Downloading DataUploader

Use the following link to download DataUploader: https://download.uipath.com/ProcessMining/versions/2.0.3/DataUploader/process-mining-data-uploader.exe

Loading data using DataUploader

Follow these steps to load data using DataUploader.

1. Download the DataUploader.

2. Create a folder and copy the input files the that you want to upload using DataUploader in this folder.

3. Open a Windows Powershell.

4. Navigate to the location where you downloaded the DataUploader.

5. Run one of the following commands:

Important:

Enclose each parameter value in quotes.

process-mining-data-uploader.exe "<csv dir>" "<sas url>" "<end-of-upload api>"

process-mining-data-uploader.exe -c "<myfolder>" -s "<my-sas-uri>" -e "<my-end-of-upload-api>"

process-mining-data-uploader.exe --config-file "path\to\config\file"

A progress bar is displayed for each file upload.

Example command string

./process-mining-data-uploader.exe "data" "https://asdf.blob.core.windows.net/a28d10b0-6822-4xxy-a197-02zz39824ad4?sv=2020-06-12&si=sap-a58d10b0-6822-4ffc-a197-02ed649824ad4&sr=c&sig=WELk3bW8zpY5rL71lJ8pPPhlOq0odLNoQleY%2BL3nPqek%3D" "https://asdf.uipath.com/org/tenant/processmining_/api/v4.0/apps/a28d10b0-6822-4ffc-a197-02ed39824ad4/ingest/authenticated?AppSecretToken=A5B221CD00E77F02677G36H572IJKLM61N2440320O44666PQ4RS3T0U81410VW" "44"

Troubleshooting

Maximum file size

The maximum file size (per file) is 5TB.

Number of retries

If the upload fails, for example, due to an unstable connection, DataUploader will retry the upload up to 4 times. If the data upload was not successful after 5 times, a report is created.

Warning:

Under certain network configurations DataUploader fails to reach the end-of-upload API due to a self signed certificate, and you will get the error such as UNABLE_TO_GET_ISSUER_CERT_LOCALLY.

This can be resolved by pointing DataUploader to your organization's Certificate Authority (CA) certificate.

Follow these steps.

1. Create an environment variable called NODE_EXTRA_CA_CERTS.

2. Set the value of the new environment variable to the file path of your CA certificate.

In case some of the files have failed the upload, you can restart DataUploader using the same command and with same files.

Calling End of upload API

What if my files are uploaded, but end of upload endpoint is not called?

Occasionally, it can happen that End of upload API is not called successfully, even after all files are uploaded. If you do not see following log after files are uploaded from data uploader, it means that End of upload API was not called successfully:

End-of-upload API reached successfully, data transformation started.

In this case, you can copy and run the following code in a PowerShell session to call the End of upload API.

$endOfUploadUri = "*******"
$delimiter = 9 # you can change this to the ascii of the character of your choice
$fullUri = "$endOfUploadUri&dataLoadOptionsJson=%7B%22UseSampleData%22%3Afalse%2C%22FileFormatOptions%22%3A%7B%22FieldDelimiter%22%3A%22$delimiter%22%7D%7D"
$response = Invoke-WebRequest -Uri $fullUri -Method POST
Write-Output $response.StatusCode

Certificates

Warning:

Under certain network configurations, DataUploader fails to reach the end-of-upload API due to a self-signed certificate, and you will get an error such as UNABLE_TO_GET_ISSUER_CERT_LOCALLY.

You can fix this by providing DataUploader with the valid certificate chain.

Follow these steps to provide DataUploader with the valid certificate chain.

1. Go to the Process Mining Portal and from the Process apps tab, open the process app.

2. Navigate to the address bar in the web browser and select the View site information icon at the left in the address bar.

   
   

   

   
   

3. Select Connection is secure, and then select Certificate is valid to display the certificate.

   
   

   

   
   

4. Go to the Details tab, and expand the Certificate Hierarchy.

5. Select the cloud.uipath.com entry and select Export....

   
   

   

   
   

6. Select a location to save the certificate file on your computer and make sure to save it as type Base64-encoded ASCII, certificate chain (*.pem;*crt).

   The cloud.uipath.crt file will appear in the selected location.

7. Create an environment variable NODE_EXTRA_CA_CERTS and set the value of this variable to the full file path of the certificate selected in step 6 (including the folder path, file name and file extension)

   1. Open the Windows System Properties.

   2. Go to the Advanced tab.

   3. Select Environment Variables.

   4. Select New to add a new System variable.

      
      

      

      
      

8. Initiate a new PowerShell session to ensure that the environment variable is recognized by the DataUploader.