activities
latest
false
UiPath logo, featuring letters U and I in white

UI Automation Activities

Last updated Dec 4, 2024

v23.10

v23.10.13

Release date: August 7, 2024

Bug fixes

  • When indicating in a Java dialog, the dialog was closing when the Java application was used as a browser extension.

v23.10.12

Release date: June 7, 2024

Bug fixes

  • When navigating to the second page of a Kendo Angular table, the Extract Table Data activity did not successfully scrape the data.
  • Fixed a Java application pop-up issue occurring starting with package version 21.10.6 that affected design time experience.

v23.10.11

Release date: April 3, 2024

Improvements

Added custom support for BasicTreeTable and JxTreeTable Java controls.
For traceability and debugging speed purposes for Computer Vision runtime analysis, DesignTime ScreenId is now sent in the header of the HTTP request.

Bug fixes

  • The OCR for Chinese, Japanese, and Korean was not working as a custom OCR for Computer Vision in Unified Target when used with a Computer Vision API key.
  • The OCR for Chinese, Japanese, and Korean was not working as a custom OCR in the Use Application/Browser activity when the project name contained Japanese characters.
  • The Appeared event type from the Application Event Trigger activity could not be monitored inside iframe HTML elements.
  • When the enabled attribute was selected, the Get Attribute activity returned a string output instead of boolean. Backwards compatibility is ensured.

v23.10.9

Release date: February 26, 2024

Bug fixes

  • A Value cannot be null exception was thrown when removing the OCR engine activity from the CV Screen Scope after having indicated a target.
  • A There is no row at position 0 exception was thrown when removing the OCR engine activity from the CV Screen Scope and the action of hovering over table cells persisted.

v23.10.8

Release date: December 21, 2023

What's new

Added support for UiPath Autopilot™ for Developers, currently available in Preview exclusively in Studio Web. Instead of building a new automation from scratch, you can now simply describe the task you want to automate using natural language and watch as Autopilot's generative AI skillset converts your description into a fully automated workflow.

Additionaly, this update contains improvements for coded automations support in Studio.

v23.10.7

Release date: December 11, 2023

Bug fixes

A System.ObjectDisposedException error was thrown when attempting to run a coded workflow that uses Object Repository.

v23.10.6

Release date: December 6, 2023

Bug fixes

  • The Java Extension couldn't be reinstalled after upgrading Studio to v2023.4 if an error occurred when trying to uninstall it. With the new changes, the Uninstall button on the Studio > Tools > UiPath Extensions page changes to Install, despite any uninstall errors. Running the install again upgrades the Java extension properly even if any errors occur after uninstalling.
  • The workaround implemented for the UiPath MV3 browser extension initialization failure in Private mode is no longer necessary starting with Edge v119, as Microsoft confirmed that the original bug has been fixed in this version. The workaround now only applies for Edge versions between 115 and 118.

v23.10.3

Release date: October 23, 2023

Browser

When the browser closes unexpectedly or when it's closed during Windows shut down, the Restore pages pop-up window is displayed. This behavior can break the automation if no specific logic is implemented on browser opening (for example, checking if the pop-up window appears and closing it with a Click activity). We implemented a change that prevents the Restore pages pop-up window from appearing when opening the browser.

Automating multiple browser instances running with multiple user profiles at the same time is now supported. This feature allows users to design automations with multiple browser instances opened under different profiles for Chrome, Edge, and Firefox.

You can now install the Firefox, RDP, and VMware extensions directly from UiPathRemoteRuntime.msi.

UI Automation Modern

The following activities have been hidden from the Modern experience:

  • OCR Text Exists
  • Click OCR Text
  • Double Click OCR Text
  • Hover OCR Text
  • Find OCR Text Position

To see these activities in the Activities panel, select the Classic filter.

Trigger-based attended automation

For situations in which you need to identify the type of native events trigged when you interact with certain UI elements in an application, we introduced a new tool integrated with UI Explorer, namely Event Inspection Tool, which can be used for configuring the Application Event Trigger activity. The tool captures and monitors all the selected event types, enabling you to analyze the actions performed and, if applicable, diagnose issues.

We've added support for many events through the Application Event Trigger activity:
  • HTML events:
    • Download changed
    • Tab activated
    • Tab created
    • Tab removed
    • Tab updated
  • WEBCTRL events:
    • Selection change
  • UI-based events:
    • Appeared
    • Disappeared
    • Text changed
    • Focus gained
    • Focus lost
    • Invoked
    • Selected
    • Deselected
    • Toggled

We also added support for WEBCTRL based native events in native embedded browsers (such as Slack, Discord, or Electron test app).

The Application Event Trigger activity can now be used with coded workflows. As a result, we also introduced a way to monitor multiple target elements with a single Application Event Trigger and allow the number of monitored targets to be dynamic (the monitored targets may change after design time). Prior to this, a triggered workflow with an Application Event Trigger only monitored a single target element, requiring you to create a triggered workflow for each target element you wanted your automation to monitor.

SAP

We added support for the following SAP WebGUI elements: SAP Menu items (using the Select Item activity), SAP Toolbar buttons (using the Click Toolbar Button activity), and SAP Trees (including stable identification of the Trees and reliable automation across all browsers and all SAP FIORI versions). Note than you can use the existing capabilities to automate various other SAP WebGUI elements, such as ABAP lists, Context menus, Pictures, Textedit controls, Generic Object Services, Dialog boxes, and Table controls.

Multi-monitor scaling awareness is now supported. This allows a multi-monitor setup where each monitor has a different scaling percentage and can move the SAP window freely across the monitors. To support this feature, a new project setting is available in Studio: Project Settings > UI Automation Classic > SAP > Enable additional scaling.
We made some improvements to the SAP Table Cell Scope activity. Filtering now includes content for table cells (for columns and rows). Before, the identification of columns in an SAP table was based on their display name, causing issues when a table had several columns with similar names. Now, these scenarios are automatically identified and you can select other attributes instead, such as ColumnName. Additionally, you can filter on columns and rows to use any other property to identify elements. Two property names were renamed to support this improvement: Column Name became Column Name / Filter, and Row Number became Row Number / Filter.

General improvements

We added support for automating legacy versions of Oracle Siebel CRM Applications (e.g. 7 or 8).

We added support for the CTable component of the Clarity Java application.

Added the Copy to Clipboard option for all UI Automation dialog messages.

We improved the error messages displayed for nested remote sessions.

Computer Vision

In this version of the activity package, we optimized core issues by accounting for each targeting method's relevancy in terms of targeting power and resilience to UI changes and we addressed the main limitations in the 23.4.x. As a result, UI Automation Table Extraction is now Computer Vision boosted when indicating full tables and table extraction is also enabled for pure Computer Vision scenarios; you can change the OCR engine as you can now select a custom OCR engine besides the embedded UiPath Screen OCR; Computer Vision is enabled by default in scopeless activities further boosting fully loaded targets (having both selectors and Computer Vision descriptors); and we optimized the experience of indicating UI elements by updating the reconciling logic between Driver and Computer Vision regarding bounding boxes. We have thus continued to pursue our vision of a unified UI Automation user experience where both pure Computer Vision and combined Computer Vision plus Driver powered automations can be created without the need for the user to be aware of the intricacies of each targeting method. More specific details can be found below.

A new hierarchical concept is now applied to Unified Target, taking into account each targeting method and its relevancy in terms of targeting power and resilience to UI changes. Hence, selectors are now considered the primary targeting method, Computer Vision is a secondary targeting method, and Image is a tertiary targeting method (and is now disabled by default). Before this, all targeting methods were trying to identify the target element simultaneously and whichever targeting method was able to find it first was used. Also, a new setting, Wait for primary targeting method until timeout, has been added in Project Settings > UI Automation Modern. The setting is set to True by default and allows the primary targeting method to consume its full timeout until any fallback targeting method can be used.

Computer Vision is now integrated in the Table Extraction wizard. You can now simply indicate full tables without having to indicate a relevant table cell first. Since Computer Vision is able to see entire tables, this functionality is now also used at design-time to correct faulty selector-based extraction when indicating a full table. At runtime, however, we don't have a Computer Vision fallback mechanism in place just yet when selector-based extraction has issues extracting full tables. You can extract pure Computer Vision tables as well without having to switch to a CV Screen Scope activity or a CV Extract Table activity for scenarios where selectors are not available at all, such as remote desktops without the remote run-time installed. On top of that, you can now also extract tables spanning across multiple pages in pure Computer Vision scenarios by indicating the Next button in the wizard.

Using the same logic as in the CV Extract Table activity, extracting scrollable tables in pure Computer Vision scenarios is now also possible in the Table Extraction wizard, as well as extracting scrollable text using CV Get Text. The Log Table Scroll Data project setting (Project Settings > Computer Vision > Robot Logging) has been renamed to Log Scrollable Content Extraction Data and it now covers Computer Vision Table Extraction, Table Extraction for scrollable tables in pure Computer Vision scenarios, and CV Get Text with scrollable text content.

In the Use Application/Browser activity, in addition to the Computer Vision embedded OCR engine, namely UiPath Screen OCR, you can now use a custom OCR engine to enhance the text extraction in Computer Vision when used in Unified Target. The OCR engine can be changed in each Use Application/Browser activity by going to the activity card > choosing the hamburger menu that appears next to the informative screenshot after indicating an UI element > selecting OCR Engine > Custom OCR Engine. When this option is selected, a panel is added in the activity card where you can drop a different OCR engine activity than the embedded one. This only affects the Computer Vision targets, as the rest of the OCR activities still use the default OCR set in Project Settings > OCR section.

As the happy path is having fully loaded targets, Computer Vision is now auto-enabled for indicating in scopeless activities (that can be used outside an Use Application/Browser scope) when it is also enabled in Project Settings.

When Computer Vision is enabled as a targeting method in Project Settings, the App/Web Recorder now captures Computer Vision descriptors as well.

The Computer Vision toggle state (enabled/disabled) in the Selection Options helper is now stored and persists throughout all the indicate sessions in the same Use Application/Browser scope activity. Every new Use Application/Browser scope activity begins with the default Computer Vision project setting for each technology (Project Settings > UI Automation Modern > Targeting methods - Desktop applications / Web browsers / Java / SAP > Computer Vision).

We adapted our reconciling logic between Driver and Computer Vision regarding bounding boxes by matching the two targets to avoid indicating a Computer Vision-only target and to favor fully loaded targets. Selecting Computer Vision-only targets should now be reserved for scenarios like remote desktop with no access to selectors.

Strict selector failures are now displayed in the logs as warnings alongside ten suggestions of the closest selector matches that can be used to replace the faulty selectors. This feature is set to True by default in Project Settings > UI Automation Modern > Show closest matches in the 'Element not found' error message.

The Computer Vision screenshot no longer needs an initial delay, so now the start of an indicate session is faster.

UI Automation APIs

If you prefer to code when building automations, you can now achieve this using coded automations. Coded automations support UiPath services, such as UI Automation activities, and allow you to create hybrid automations, by integrating them with low-code automations.

Visit the UI Automation APIs section to browse through the available APIs and Coded Automations Introduction to learn how to create coded automations.

Bug fixes

  • Fixed the broken Active Accessibility support for web pages containing IFrame elements or PDFs in Chrome and Edge version 117, which caused <ctrl /> selectors to fail.
    Also added the option to pass extra command-line arguments to the browser opened using the Open Browser / Use Browser activities. This can be done by setting the appropriate environment variable using the Set Environment Variable activity:
    • UIPATH_EXTRA_CMD_ARGS_CHROME
    • UIPATH_EXTRA_CMD_ARGS_EDGE
    • UIPATH_EXTRA_CMD_ARGS_FIREFOX

    These variables can be used to force-enable the Active Accessibility support in Chrome and Edge to avoid issues like the broken Chrome/Edge Active Accessibility support from versions 114 and 117.

    For Chrome, set the environment variable UIPATH_EXTRA_CMD_ARGS_CHROME with the value --force-renderer-accessibility=complete.
    For Edge, set the environment variable UIPATH_EXTRA_CMD_ARGS_EDGE with the value --force-renderer-accessibility=complete.
  • MV3 browser extensions failed to initialize when using Microsoft Edge (versions 115 and newer) in Private mode. We have implemented a workaround for this issue that affected all MV3 extensions, not just the UiPath Extension for Microsoft Edge.
  • We addressed some extension initialization issues affecting the Open Browser and Use Browser activities by adding six UI Automation Classic – Browser project settings: Extra arguments for OpenBrowser Chrome/Edge, Custom executable path for OpenBrowser Chrome/Edge, and Enable retry for OpenBrowser Chrome/Edge.
  • The Table Cell Scope activity didn't recognize column names when SAP WinGUI was set in Japanese.
  • Data Scraping returned empty cells for Salesforce reports.
  • Support for Chrome/Edge running in App Mode is now available in the self-hosted MV2 browser extension installed via Group Policy Offline.
  • When installing extensions with Admin rights, UiPath.Common is now automatically deployed to %ProgramFiles(x86)%\UiPath\UiPath.Common. When installing extensions without Admin rights, UiPath.Common is now automatically deployed to %LocalAppData%\Programs\UiPath\UiPath.Common.
  • We added support for web pages using the ServiceNow Framework.

  • The tableCol tag in the selector prevented some table elements from being identifiable.
  • You can now automate applications running under a different Windows User (other than the one running UiPath Robot/Studio), without having to run Robot/Studio as administrator.
  • In a cross-platform Use Browser activity, the browser connection failed or became unresponsive when more than one tab was opened at design time on the serverless machine where the connection got confirmed, the user was logged out of Studio, the user deleted her/his personal workspace in Orchestrator, or when encountering a communication error with Orchestrator.
  • Adding a selector from an activity to Object Repository failed with the "The operation was canceled" error message when the reuse search took more than three seconds.
  • The Use Application/Browser activity threw an exception at design time in StudioX due to the asterisk indicating a property field is required.
  • The Send Hotkey activity caused the .xaml file to be shown as unsaved.
  • Image screenshot was still captured even when Image was disabled as a targeting method in Project Settings.
  • The You Are Calling The Computer Vision Server Too Often [429] error message displayed when reaching the Computer Vision rate limit has been changed to Error: Rate limit exceeded. If you're on a free tier, consider upgrading to paid license. If already on a paid license, please contact UiPath for further upgrade options.
  • The modern Get Text activity did not extract text in pure Computer Vision scenarios.

Was this page helpful?

Get The Help You Need
Learning RPA - Automation Courses
UiPath Community Forum
Uipath Logo White
Trust and Security
© 2005-2024 UiPath. All rights reserved.