Performance Best Practices

The following page describes the recommended requirements for large-scale production environments and provides a set of best practices aimed at improving their performance.

Note: For more details on the minimum requirements to run Orchestrator, see Hardware Requirements and Software Requirements pages.

Recommended Requirements for Large Scale Production Environments

To run 10K Attended Robots or 1K Unattended Robots, you need:

• An F5 load balancer
• Orchestrator - at least 6 nodes that run on machines with 8 CPU Cores and 16 GB RAM
• Robots - machines with 4 CPU Cores and 16 GB RAM

• SQL Server - machines with 4 sockets / 16 CPU Cores (default 8 sockets/16 CPU Cores) and 64 GB RAM

  Note: To reduce SQL Server allocation contention in a highly concurrent environment, make sure you employ an optimal number of tempdb data files that have equal sizing.

Orchestrator Changes for Improved Performance

This section lists a series of adjustments you can make to Orchestrator's UiPathOrchestrator.dll.config file in order to improve performance in a large scale production environment.

Enable Redis

<appSettings>
  <add key="LoadBalancer.UseRedis" value="true"/>
  <add key="LoadBalancer.Redis.ConnectionString" value="your.redis.cache.windows.net:6379,password=***"/>
</appSettings>

Specify a Large Max Pool Size in the Connection String

<connectionStrings>
  <add name="Default" providerName="System.Data.SqlClient" connectionString="Data Source=<SQLServer>;Initial Catalog=<dbName>;User ID=<username>;Password=***;Max Pool Size=1000" />
</connectionStrings>

Write Robot Logs to Elasticsearch and Use AsyncWrapper

• Add or modify the logging target for Elasticsearch using the following parameters.

  xml <nlog> <targets> <target name="robotElasticBuffer" xsi:type="AsyncWrapper" overflowAction="Grow" queueLimit="10000" batchSize="1000" timeToSleepBetweenBatches="1"> <target xsi:type="ElasticSearch" name="robotElastic" uri="elastic.example.com:9200" requireAuth="true" username="***" password="***" index="${event-properties:item=indexName}-${date:format=yyyy.MM}" documentType="logEvent" includeAllProperties="true" layout="${message}" excludedProperties="agentSessionId,tenantId,indexName" /> </target> </targets> </nlog>

• Add or modify the robot logs rule to write logs only to the target shown in the previous example. This automatically disables the default Database target.

  xml <nlog> <rules> <logger name="Robot.*" ruleName="primaryRobotLogsTarget" final="true" writeTo="robotElasticBuffer" /> </rules> </nlog>

• Configure Orchestrator to read logs from ElasticSearch.

  <appSettings>
     <add key="Logs.RobotLogs.ReadTarget" value="robotElasticBuffer" />
  </appSettings>

Use a High Throughput Storage

You can opt for Azure Blob Storage for this purpose.

<appSettings>
  <add key="Storage.Type" value="Azure" />
  <add key="Storage.Location" value="DefaultEndpointsProtocol=https;AccountName=yourBlob;AccountKey=yourKey;EndpointSuffix=core.windows.net" />
</appSettings>

Disable Webhooks and Ledger

If you don’t use the Webhooks feature, you can disable it.

<appSettings>
  <add key="Webhooks.Enabled" value="false" />
  <add key="Ledger.Enabled" value="false"/>
  <add key="Webhooks.LedgerIntegration.Enabled" value="false" />
</appSettings>

Disable Maintenance Mode

If you don’t use Maintenance Mode, you can disable it.

<appSettings>
  <add key="MaintenanceMode.Enabled" value="false" />
</appSettings>

Increase the Caching Time of Application Settings

You can configure Orchestrator to use some in-memory caches that reduce the number of calls to Redis. For instance, configuring them to expire every 30 minutes means that, once you make a change to an application-level setting, it may not be reflected until after the 30-minute interval.

The expiry time you set here can vary depending on what delay you can tolerate, but even a 5-minute caching time can greatly improve performance.

<appSettings>
  <add key="Caching.MultiTier[AbpApplicationSettingsCache]" value="Memory" />
  <add key="Caching.MultiTier[AbpApplicationSettingsCache].ExpiresAfter" value="0:30:00" />
  <add key="Caching.MultiTier[AbpTenantSettingsCache]" value="Memory"/>
  <add key="Caching.MultiTier[AbpTenantSettingsCache].ExpiresAfter" value="0:30:00"/>
  <add key="Caching.MultiTier[AbpZeroTenantFeatures]" value="Memory"/>
  <add key="Caching.MultiTier[AbpZeroTenantFeatures].ExpiresAfter" value="0:30:00"/>
  <add key="Caching.MultiTier[AbpZeroRolePermissions]" value="Memory"/>
  <add key="Caching.MultiTier[AbpZeroRolePermissions].ExpiresAfter" value="0:30:00"/>
  <add key="Caching.MultiTier[LicenseKeys]" value="Memory"/>
  <add key="Caching.MultiTier[LicenseKeys].ExpiresAfter" value="0:30:00"/>
  <add key="Caching.MultiTier[TenantLicense]" value="Memory"/>
  <add key="Caching.MultiTier[TenantLicense].ExpiresAfter" value="0:30:00"/>
</appSettings>

Increase MinWorkerThreads Value

Setting a larger value for MinWorkerThreads helps with ramp-up scenarios.

<appSettings>
  <add key="ThreadPool.MinWorkerThreads" value="50" />
</appSettings>

Tune Quartz for Very High Throughput

This step is necessary only if you have a very high number of triggers firing, e.g. on the order of 2,000 per minute.

<quartz>
  <add key="quartz.scheduler.batchTriggerAcquisitionMaxCount" value="15" />
  <add key="quartz.threadPool.threadCount" value="15" />
  <add key="quartz.jobStore.misfireThreshold" value="3600000" />
  <add key="quartz.jobStore.clusterCheckinInterval" value="60000" />
  <add key="quartz.jobStore.clustered" value="true" />
</quartz>

Disable IP to DNS Resolving in Audit

If you are not in a corporate network or do not need to see the DNS names of IPs recorded in Audit, you can disable the resolving.

<appSettings>
  <add key="Audit.UseDnsResolving" value="false" />
</appSettings>

Switch to a Faster Strategy for Fetching Groups Membership

Note: Directory.ActiveDirectory.GroupMembershipFetchStrategy and Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours only apply to the legacy AD adapter. You should use these settings only if you upgraded from an Orchestrator version that had AD enabled to 2021.10 or later.

To set Directory.ActiveDirectory.GroupMembershipFetchStrategy to TokenGroups in the [identity].[Settings] table, run:

SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.GroupMembershipFetchStrategy' and [PartitionId]=1
IF @@ROWCOUNT>0
    UPDATE [identity].[Settings] SET [Value]='TokenGroups' WHERE [Key]='Directory.ActiveDirectory.GroupMembershipFetchStrategy' and [PartitionId]=1
ELSE
    INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.GroupMembershipFetchStrategy', 'TokenGroups', 1)

To set Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours to 4 hours in the [identity].[Settings] table, run:

SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours' and [PartitionId]=1
IF @@ROWCOUNT>0
    UPDATE [identity].[Settings] SET [Value]='4' WHERE [Key]='Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours' and [PartitionId]=1
ELSE
    INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.AuthorizationGroupsCacheExpirationHours', '4', 1)

Adjust AD Domain Cache Expiration Time

Note: Directory.ActiveDirectory.CacheExpirationSeconds only applies to the legacy AD adapter. You should use this setting only if you upgraded from an Orchestrator version that had AD enabled to 2021.10 or later.

To control the AD domain cache expiration time, you can use the Directory.ActiveDirectory.CacheExpirationSeconds setting in the [identity].[Settings] table. The default value of this setting is 43200 seconds (12 hours). To disable AD domain caching, you must set this setting to 0.

To set this setting to 7 days, run:

SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
IF @@ROWCOUNT>0
    UPDATE [identity].[Settings] SET [Value]='604800' WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
ELSE
    INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.CacheExpirationSeconds', '604800', 1)

To disable AD domain caching, run:

SELECT * FROM [identity].[Settings] WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
IF @@ROWCOUNT>0
    UPDATE [identity].[Settings] SET [Value]='0' WHERE [Key]='Directory.ActiveDirectory.CacheExpirationSeconds' and [PartitionId]=1
ELSE
    INSERT INTO [identity].[Settings] ([Key], [Value], [PartitionId]) VALUES ('Directory.ActiveDirectory.CacheExpirationSeconds', '0', 1)

Disable SignalR for Robot License Change Events

During an aggressive ramp-up, the license acquire events would put significant pressure on the SignalR Redis backplane. To mitigate that, the event can be disabled, which results in the licensing UI no longer being updated in real time. However, when you have 100k robots, that screen updating in real time is of little value.

<appSettings>
  <add key="Scalability.SignalR.Browser.RobotLicenseChangeEventEnabled" value="false" />
</appSettings>

Enable NuGet Package Caching

This change ensures the server-side caching of the package contents, and it can prove useful when the file share cannot keep up with the package download throughput.

<<add key="NuGet.Caching.Enabled" value="true" />
<add key="NuGet.Caching.MaxCacheSize" value="104857600" /><!-- (100 MiB) -->
<add key="NuGet.Caching.MaxPackageSize" value="4194304" /><!-- (4 MiB) -->
<add key="NuGet.Caching.AbsoluteExpirationRelativeToNow" value="00:10:00" /><!-- (1 min) -->

Adding the NuGet.Caching.Enabled setting should be enough to enable the feature. You only need to adjust the rest of the settings displayed in the example above if you want to change the defaults, e.g. increase the size of the cache or package size limits.

Enable Local Cache in Azure PaaS Deployments

Set the following configuration as slot sticky on the production slot of the Orchestrator App Service.

WEBSITE_LOCAL_CACHE_OPTION = Always
WEBSITE_LOCAL_CACHE_SIZEINMB = 1000

Configure Slot Warm-Up in Azure PaaS Deployments

Set the following configuration on both the production and hotswap deployment slots of the Orchestrator App Service. During deployment, this will warm up the application, minimizing the performance hit caused by redirecting traffic to a cold app.

WEBSITE_SWAP_WARMUP_PING_PATH = /api/status
WEBSITE_SWAP_WARMUP_PING_STATUSES = 200

Use Azure SignalR Service in Azure PaaS Deployments

Using Azure SignalR Service in an Azure deployment can significantly reduce the load on the web servers. See Scalability.AzureSignalR.ConnectionString for more details on the setting.

<appSettings>
  <add key="LoadBalancer.UseRedis" value="true"/>
  <add key="Scalability.AzureSignalR.ConnectionString" value="**myAzureSignalRConnectionString" />
</appSettings>

Adjust the Flush Interval and Size for the SubmitLogs Endpoint

This change helps you control the timeframe batch window for collecting logs from the executor process before calling the SubmitLogs API endpoint. Enable the following settings in UiPath.Orchestrator.dll.config and configure them according to your needs:

• RobotsLogs.Flush.Interval

  <add key="RobotsLogs.Flush.Interval" value="0" />

For more on this setting, see RobotsLogs.Flush.Interval.

• RobotsLogs.Flush.BatchSize

  <add key="RobotsLogs.Flush.BatchSize" value="100" />

For more on this setting, see RobotsLogs.Flush.BatchSize.

Identity Server Changes for Improved Performance

The following changes to Identity Server's AppSettings.Production.json file should result in improved performance in large scale production environments.

Specify a Large Max Pool Size in the Connection String

"ConnectionStrings": {
    "DefaultConnection": "Data Source=dbServer;Initial Catalog=UiPath_is;User ID=username;Password=****;Max Pool Size=1000"
  }

Enable Redis

"LoadBalancerSettings": {
      "RedisConnectionString": "yourRedis:6379,password=****",
      "UseRedis": "true"
    }

UI Configuration Changes for Improved Performance

Orchestrator's UI allows you to take the following steps in order to improve performance in large scale production environments:

• Disable personal workspace;
• Disable Allow web login for robot users;
• Assign Robot and Automation User roles to robot users.