- Overview
- UI Automation
- Applications and technologies automated with UI Automation
- UI-ANA-016 - Pull Open Browser URL
- UI-ANA-017 - ContinueOnError True
- UI-ANA-018 - List OCR/Image Activities
- UI-DBP-006 - Container Usage
- UI-DBP-013 - Excel Automation Misuse
- UI-DBP-030 - Forbidden Variables Usage In Selectors
- ST-DBP-021 - Hardcoded Timeout
- UI-PRR-001 - Simulate Click
- UI-PRR-002 - Simulate Type
- UI-PRR-003 - Open Application Misuse
- UI-PRR-004 - Hardcoded Delays
- UI-REL-001 - Large Idx in Selectors
- UI-SEC-004 - Selector Email Data
- UI-SEC-010 - App/Url Restrictions
- UI-USG-011 - Non Allowed Attributes
- SY-USG-013 - Private File Invoke
- SY-USG-014 - Incorrect Execution Template Placeholders
- UiPath Browser Migration Tool
- App/Web Recorder
- Advanced descriptor configuration
- UI elements extraction
- Table Extraction
- Linux Robots
- UI Automation Browser Connection
- Computer Vision Recorder
- Activate
- Anchor Base
- Attach Browser
- Attach Window
- Block User Input
- Call Transaction
- Callout
- Check
- Click
- Click Image
- Click Image Trigger
- Click OCR Text
- Click Picture on Screen
- Click Text
- Click Toolbar Button
- Click Trigger
- Close Application
- Close Tab
- Close Window
- Context Aware Anchor
- Copy Selected Text
- Element Attribute Change Trigger
- Element Exists
- Element Scope
- Element State Change Trigger
- Expand ALV Hierarchical Table
- Expand ALV Tree
- Expand Tree
- Export UI Tree
- Extract Structured Data
- Find Children
- Find Element
- Find Image
- Find Image Matches
- Find OCR Text Position
- Find Relative Element
- Find Text Position
- Get Active Window
- Get Ancestor
- Get Attribute
- Get Event Info
- Get From Clipboard
- Get Full Text
- Get OCR Text
- Get Password
- Get Position
- Get Source Element
- Get Text
- Get Visible Text
- Go Back
- Go Forward
- Go Home
- Google Cloud Vision OCR
- Hide Window
- Highlight
- Hotkey Trigger
- Hover
- Hover Image
- Hover OCR Text
- Hover Text
- Image Exists
- Indicate On Screen
- Inject .NET Code
- Inject JS Script
- Invoke ActiveX Method
- Key Press Trigger
- Load Image
- Maximize Window
- Microsoft Azure Computer Vision OCR
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Minimize Window
- Monitor Events
- Mouse Trigger
- Move Window
- Navigate To
- OCR Text Exists
- On Element Appear
- On Element Vanish
- On Image Appear
- On Image Vanish
- Open Application
- Open Browser
- Read Statusbar
- Refresh Browser
- Replay User Event
- Restore Window
- SAP Login
- SAP Logon
- SAP Session Attributes Change Trigger
- Save Image
- Select Dates in Calendar
- Select Item
- Select Menu Item
- Select Multiple Items
- Send Hotkey
- Set Clipping Region
- Set Focus
- Set Text
- Set To Clipboard
- Set Web Attribute
- Show Window
- Start Process
- System Trigger
- Table Cell Scope
- Take Screenshot
- Tesseract OCR
- Text Exists
- Tooltip
- Type Into
- Type Secure Text
- Use Foreground
- Wait Attribute
- Wait Element Vanish
- Wait Image Vanish
- Application Event Trigger
- Check/Uncheck
- Check App State
- Check Element
- Click
- Click Event Trigger
- Close Popup
- Drag and Drop
- Extract Table Data
- For Each UI Element
- Get Attribute
- Get Browser Data
- Get Text
- Get URL
- Go to URL
- Highlight
- Hover
- Inject Js Script
- Keyboard Shortcuts
- Keypress Event Trigger
- Mouse Scroll
- Navigate Browser
- Select Item
- Set Browser Data
- Set Runtime Browser
- Set Text
- Take Screenshot
- Type Into
- Use Application/Browser
- Perform browser search and retrieve results using UI Automation APIs
- Web Browsing
- Find Images
- Click Images
- Trigger and Monitor Events
- Create and Override Files
- HTML Pages: Extract and Manipulate Information
- Window Manipulation
- Automated List Selection
- Find and Manipulate Window Elements
- Manage Text Automation
- Load and Process Images
- Manage Mouse Activated Actions
- Automate Application Runtime
- Automated Run of a Local Application
- Browser Navigation
- Web Automation
- Trigger Scope Example
- Computer Vision Local Server
- Mobile Automation
- Release notes
- Project compatibility
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Install App
- Manage Current App
- Manage Other App
- Open DeepLink
- Open URL
- Mobile Device Connection
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Take Screenshot Part
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Swipe
- Tap
- Type Text
- Draw Pattern
- Terminal
Advanced descriptor configuration
For more information on using the Selection Screen's more advanced features, check out the video below.
Indicating a target may produce duplicates, which are other UI elements that are perceived as extremely similar to the target UI element and might cause the automation to fail at runtime. When duplicates are found, they are highlighted in yellow and have a label above them containing a letter which corresponds to the targeting method that found each duplicate:
- S - Strict selector
- F - Fuzzy selector
- CV - Computer Vision
- I - Image
One of the best ways to avoid duplicates is indicating an anchor.
To create a reliable descriptor, it is recommended to indicate an anchor after the target is indicated. You can indicate up to three anchors for a target. Anchors can be found in several scenarios, as described below:
After indicating a target, an attempt is made to auto-identify an anchor for it.
In this section, you can find out more technical details about the method in which this search is performed.
Once a target is indicated, the type of the indicated UI element is identified. The following elements and directions are searched for:
- Button - inside of the element.
- Check box - to the right of the element.
- Drop-down - to the left and above the element.
- Input box - to the left and above the
element. In the case of browser automation, the label is also searched inside the
input box, looking for the
AreaLabeledBy
attribute. If this label is not identified, an element with thetag=label
andfor='id of the input box'
is searched for. - Radio button - to the right of the element.
- Image - to the right and below the element.
Depending on the type of element that is identified, a corresponding anchor is searched for the element automatically.
An automatically identified anchor is in the form of a text element that is always present near the target element and thus can be used to identify the element at runtime with more precision.
If none of the above directions provide results, a generic case is used, which searches in the element hierarchy for a suitable anchor.
In this situation, you can manually select an anchor for your target element. When hovering another element, it is highlighted blue and connected with a line to the target you have selected. To select the anchor you want, you can simply click the text you want to use as an anchor.
You can also edit the a specific element (be it target or anchor) by accessing the menu that is displayed when hovering over the target element. This menu has multiple options:
The button closes the indicate process and saves the changes you have made.
The button enables you to target elements regardless of their specific text content. This can be helpful when trying to select the first element in a search list when you don't know exactly what the list is going to be populated with, or when interacting with a dynamic element that changes its label due to various reasons. Using this feature when indicating a target uses only the Fuzzy selector method for identification, and strips down the Fuzzy selector to make sure the element is still targeted regardless of label changes.
By clicking the button, you can select an adjacent UI element that acts as an additional anchor. You can indicate up to three anchors for any target.
The button expands the target section in the Selection Options, enabling you to configure the targeting methods used for the current descriptor.
The button deletes the target you have indicated and lets you choose another one.
The Options section can be expanded to reveal the following options:
- Auto-apply changes - When selected, auto-applies changes to target and anchor elements.
- Responsive websites - When selected, enables the anchor to automatically move from left to the top of the target, or from top to the left of the target, thus letting you automate websites that are responsive.
- Element visibility - Checks
whether the UI element is visible or not. There are three options available from the
drop-down menu:
- None - Does not check for visibility.
- Interactive (for Fuzzy selector) - Default option. Checks if the element is potentially visible, ignoring page scroll and obstructions by other apps, or the fact that the application is minimized. This check is useful when trying to ensure that you are not targeting invisible elements that exist in the DOM but are hidden.
- Fully visible - Ensures that the UI element is visible on the screen.
The Window Selector section can be expanded to view and edit the selector of the indicated target application. You can also use the Open in UI Explorer button to edit the window selector in the UI Explorer.
The under-the-hood process of finding an UI element uses several targeting methods:
- The primary targeting method is represented by selectors: Strict selector or Fuzzy selector.
- The secondary targeting method is Computer Vision.
- The tertiary targeting method is Image (disabled by default).
The icon is displayed next to the leading targeting method at design time, simulating the scenario at runtime.
Default project settings for targeting methods across technologies
The below table displays the default project settings for targeting methods across different technologies.
Targeting methods | Desktop applications | Web browsers | Java | SAP |
---|---|---|---|---|
Strict selector | ||||
Fuzzy selector | ||||
Computer Vision | ||||
Image |
Project setting: Wait for primary targeting method until timeout
The Wait for primary targeting method until timeout project setting (Project Settings > UI Automation Modern), which is enabled by default in new projects starting with activity package version 23.10.3 and which can be enabled manually for projects created prior to this activity package version, offers the primary targeting method some extra time to consume its full timeout until any fallback targeting method can be used.
If one of the primary targeting methods manages to find the UI element until the timeout expires, then it's going to be used as the method to interact with the UI element. Strict selector is the targeting method that first searches for the UI element, followed shortly by the Fuzzy selector. Also, Strict selector has priority over Fuzzy selector.
When the UI element is not found by the primary targeting methods until the timeout expires, so if after all the retries within the allocated timeout neither Strict selector nor Fuzzy selector find the UI element, only then the secondary and the tertiary targeting methods start searching for the UI element within a timeout of their own.
This project setting also brings more predictability and visibility into selector related issues so that you can have better control over your UI elements targeting, while also having the solid fallback that Computer Vision provides.
If you disable this project setting, then all the targeting methods compete in a single timeout to find the UI element according to the hierarchy: a target found by lower ranking targeting method is used only if a higher ranking targeting method fails to do so.
Project setting: Show closest matches in the 'Element not found' error message
The Show closest matches in the 'Element not found' error message project setting (Project Settings > UI Automation Modern) is enabled by default in new projects starting with activity package version 23.10.3 and it can be enabled manually for projects created prior to this activity package version. When this setting is enabled, if a selector fails, a warning is displayed in the logs showcasing the method used to find the UI element together with ten suggestions of the closest selector matches that can be used to replace the faulty selector.
It is not recommended to manually edit the descriptor, as this entire process is thought out to be seamless for the user and should provide good results without the need for advanced configuration. Still, if you want to configure more advanced features for your descriptor, you can enable, disable, and edit the targeting methods.
Selectors are XML fragments that contain the attributes of UI elements and their parents. Selectors are used for targeting UI elements on the screen, by inspecting them and their attributes, and matching them with the attributes in the selector. You can find out more on what a selector is in the UiPath ecosystem by viewing this page.
The Strict selector of the target element can be edited from its dedicated section in the Selection Options window. You can also use the Open in UI Explorer button to edit the selector in the UI Explorer.
The Strict selector field supports the use and creation of variables and arguments, from the context menu (accessible by right-clicking).
You can also select a part of the selector and use the context menu to directly convert it into a variable that is pre-populated with your selection.
You can also select an already created variable from the Variable Name drop-down menu and use it in your selector.
The Override default value check box enables you to replace the value in your predefined variable with your current selection.
The Confirm button saves the changes you have made in your selector.
The following hotkeys are available:
Ctrl
+Space
: choose variableCtrl
+Shift
+Space
: choose argumentCtrl
+K
: create variableCtrl
+M
: create argument
If there are both a variable and an argument with the same name, the variable is always defaulted to and used at runtime.
Arguments with names identical to variables are hidden from the variable selection menu in the Selection Screen.
Changes done with variables and arguments can be propagated throughout all the other instances where they are found. For example, if a variable is used in both the Strict selector and Fuzzy selector fields, changing it in one of the fields generates a message that informs you the same value was found in another field, and offering you the option to replace all instances with the new value.
Strict selectors also support wildcards, which can be used to replace zero or multiple characters in a string. You can find out more about this functionality here.
The Fuzzy selector is very similar to Strict selector, with the difference that its attributes are fuzzy matched instead of perfectly matched. This means that a Fuzzy selector can output multiple elements, as opposed to Strict selector which outputs a single element at a time.
The Fuzzy selector can be edited from its dedicated section in the Selection Options window.
The matching accuracy of the Fuzzy selector is based on the Levenshtein distance algorithm to count the similarity of selectors. The default value is 0.5 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The matching accuracy can be set within a range of 0.4 to 1 where a lower value indicates a higher degree of fuzziness, meaning the Fuzzy selector outputs more elements, and a higher value indicates a lower degree of fuzziness, meaning the Fuzzy selector outputs less elements.
Just like the Strict selector, the Fuzzy selector field supports the use and creation of variables and arguments and it also supports wildcards. For more details, refer to the information provided above.
If the indicated element contains text, the text is displayed in the input box below the Fuzzy selector edit window. If text is displayed in this input box, at runtime it must be found in the element itself to be considered a match.
The drop-down menu next to the input box has the following options:
- Text is/InnerText is/FullText is (depending on the text scraping method used) - Displays the text that is scraped; the best scraping method is automatically identified and it is recommended to use it.
- Ignore text - Ignores the text specified in the input box.
If Image automation is used for your target, you can also view the screenshot of the element, as well as adjust the image accuracy. The default value is 0.8 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The accuracy can be set within a range of 0.4 to 1. For further information, check the Image and Text Automation section.
If you want to define Image-only targets, you can use the Image region selection (F3) or draw a box selection of the area you want to use as your target/anchor while indicating.
You can also use the refresh button to manually refresh the image that you have selected, in case the image of the element changed since it was indicated.
The Image targeting method is disabled by default in Project Settings.
To enable it at project level, go to Project Settings > UI Automation Modern > Targeting methods - Desktop applications / Targeting methods - Web browsers / Targeting methods - Java / Targeting methods - SAP (depending on what type of application you want to automate) > Image > select True.
To enable the method on a specified element, the element must be valid.
If you selected the Enable Native text check box when indicating the element, the Native Text targeting method is used. This method enables you to use all the text labels that are identified with the Native method for identifying the element you want to interact with.
The text label identified through this method can be edited from its dedicated section, and you can specify whether you want the text to be case sensitive or not by using the Toggle case sensitive button.
Selecting the Edit offline option opens the selection screen in offline editing mode. In offline editing mode, you cannot add anchors, validate the descriptor, or other functionality that requires a live application.
The validation process ensures that all combinations of search methods for the target are working correctly. You can validate your selection by using the Validate button in the Selection Screen.
If you have a target and anchor pair, each of them uses all the available methods for targeting. To find the final target, all possible combinations for each of these elements are matched to elements on the screen and a score is given after the match.
After performing the validation, if there are any duplicates found, they are highlighted in the target application. If no duplicates are found, icons appear alongside the targeting methods, informing you of the state of each targeting method in the given situation. The following symbols are used:
- the targeting method that was fastest
- the method worked, but was not the fastest
- the method found duplicates and the correct element was identified by using this method alongside other targeting methods
- the method found duplicates and the correct element could not be identified by using this method
- the method failed to identify any elements