- Release notes
- UI-ANA-016 - Pull Open Browser URL
- UI-ANA-017 - ContinueOnError True
- UI-ANA-018 - List OCR/Image Activities
- UI-DBP-006 - Container Usage
- UI-DBP-013 - Excel Automation Misuse
- UI-DBP-030 - Forbidden Variables Usage In Selectors
- ST-DBP-021 - Hardcoded Timeout
- UI-PRR-001 - Simulate Click
- UI-PRR-002 - Simulate Type
- UI-PRR-003 - Open Application Misuse
- UI-PRR-004 - Hardcoded Delays
- UI-REL-001 - Large Idx in Selectors
- UI-SEC-004 - Selector Email Data
- UI-SEC-010 - App/Url Restrictions
- UI-USG-011 - Non Allowed Attributes
- SY-USG-013 - Private File Invoke
- SY-USG-014 - Incorrect Execution Template Placeholders
- Applications and technologies automated with UI Automation
- UiPath Browser Migration Tool
- App/Web Recorder
- Advanced descriptor configuration
- UI elements extraction
- Table Extraction
- Linux Robots
- UI Automation Browser Connection
- Computer Vision Recorder
- Click
- Hover
- Type Secure Text
- Send Hotkey
- Type Into
- Check
- Select Item
- Select Multiple Items
- Get Text
- Set Text
- Activate
- Highlight
- Set Focus
- Anchor Base
- Context Aware Anchor
- Find Relative Element
- Indicate On Screen
- Element Exists
- Find Element
- Wait Element Vanish
- Find Children
- Get Ancestor
- Element Scope
- Get Attribute
- Set Clipping Region
- Get Position
- Take Screenshot
- Wait Attribute
- On Element Appear
- On Element Vanish
- Click Text
- Hover Text
- Find Text Position
- Get Full Text
- Get Visible Text
- Extract Structured Data
- Text Exists
- Click OCR Text
- Hover OCR Text
- Get OCR Text
- Find OCR Text Position
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Microsoft Azure Computer Vision OCR
- Tesseract OCR
- Google Cloud Vision OCR
- OCR Text Exists
- Click Image
- Hover Image
- Find Image Matches
- Image Exists
- Find Image
- Wait Image Vanish
- On Image Appear
- On Image Vanish
- Load Image
- Save Image
- Attach Browser
- Close Tab
- Go Back
- Go Forward
- Go Home
- Inject JS Script
- Navigate To
- Open Browser
- Refresh Browser
- Set Web Attribute
- Attach Window
- Close Window
- Get Active Window
- Hide Window
- Maximize Window
- Minimize Window
- Move Window
- Restore Window
- Show Window
- Export UI Tree
- Invoke ActiveX Method
- Inject .NET Code
- Click Trigger
- Key Press Trigger
- Click Image Trigger
- System Trigger
- Hotkey Trigger
- Mouse Trigger
- Get Event Info
- Monitor Events
- Get Source Element
- Replay User Event
- Block User Input
- Use Foreground
- Element Attribute Change Trigger
- Element State Change Trigger
- Callout
- Copy Selected Text
- Get From Clipboard
- Set To Clipboard
- Close Application
- Open Application
- Start Process
- Get Password
- Tooltip
- Click Picture on Screen
- Click Toolbar Button
- Read Statusbar
- Select Dates in Calendar
- Select Menu Item
- Expand Tree
- Call Transaction
- SAP Logon
- SAP Login
- Table Cell Scope
- SAP Session Attributes Change Trigger
- Expand ALV Tree
- Expand ALV Hierarchical Table
- Use Application/Browser
- Check App State
- Check/Uncheck
- Click
- Extract Table Data
- Get Text
- Get URL
- Go to URL
- Highlight
- Hover
- Keyboard Shortcuts
- Navigate Browser
- Select Item
- Take Screenshot
- Type Into
- Get Attribute
- Get Browser Data
- Set Browser Data
- Mouse Scroll
- Inject Js Script
- Drag and Drop
- For Each UI Element
- Application Event Trigger
- Click Event Trigger
- Keypress Event Trigger
- Set Text
- Check Element
- Perform browser search and retrieve results using UI Automation APIs
- Web Browsing
- Find Images
- Click Images
- Trigger and Monitor Events
- Create and Override Files
- HTML Pages: Extract and Manipulate Information
- Window Manipulation
- Automated List Selection
- Find and Manipulate Window Elements
- Manage Text Automation
- Load and Process Images
- Manage Mouse Activated Actions
- Automate Application Runtime
- Automated Run of a Local Application
- Browser Navigation
- Web Automation
- Trigger Scope Example
- Release Notes
- Project Compatibility
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Install App
- Manage Current App
- Manage Other App
- Open DeepLink
- Open URL
- Mobile Device Connection
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Take Screenshot Part
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Swipe
- Tap
- Type Text
- Draw Pattern
Advanced descriptor configuration
For more information on using the Selection Screen's more advanced features, check out the video below.
Duplicates
Indicating a target may produce duplicates, which are other UI elements that are perceived as extremely similar to the target UI element and might cause the automation to fail at runtime. When duplicates are found, they are highlighted in yellow and have a label above them containing a letter which corresponds to the targeting method that found each duplicate:
- S - Strict selector
- F - Fuzzy selector
- I - Image
- CV - Computer Vision
One of the best ways to avoid duplicates is indicating an anchor.
Anchors
To create a reliable descriptor, it is recommended to indicate an anchor after the target is indicated. You can indicate up to three anchors for a target. Anchors can be found in several scenarios, as described below:
Anchor automatically found
After indicating a target, an attempt is made to auto-identify an anchor for it.
Anchor identification method
In this section, you can find out more technical details about the method in which this search is performed.
Once a target is indicated, the type of the indicated UI element is identified. The following elements and directions are searched for:
- Button - inside of the element.
- Check box - to the right of the element.
- Drop-down - to the left and above the element.
- Input box - to the left and above
the element. In the case of browser automation, the label is also searched
inside the input box, looking for the
AreaLabeledBy
attribute. If this label is not identified, an element with thetag=label
andfor='id of the input box'
is searched for. - Radio button - to the right of the element.
- Image - to the right and below the element.
Depending on the type of element that is identified, a corresponding anchor is searched for the element automatically.
An automatically identified anchor is in the form of a text element that is always present near the target element and thus can be used to identify the element at runtime with more precision.
If none of the above directions provide results, a generic case is used, which searches in the element hierarchy for a suitable anchor.
Manually adding an anchor
In this situation, you can manually select an anchor for your target element. When hovering another element, it is highlighted blue and connected with a line to the target you have selected. To select the anchor you want, you can simply click the text you want to use as an anchor.
Hover menu
You can also edit the a specific element (be it target or anchor) by accessing the menu that is displayed when hovering over the target element. This menu has multiple options:
The button closes the indicate process and saves the changes you have made.
The
button enables you to target elements regardless of their specific text content.
This can be helpful when trying to select the first element in a search list when
you don't know exactly what the list is going to be populated with, or when
interacting with a dynamic element that changes its label due to various reasons.
Using this feature when indicating a target uses only the Fuzzy selector method for
identification, and strips down the Fuzzy selector to make sure the element is still
targeted regardless of label changes.
By clicking the button, you can select an adjacent UI element that acts as
an additional anchor. You can indicate up to three anchors for any target.
The
button expands the target section in the Selection Options, enabling you to
configure the targeting methods used for the current descriptor.
The
button deletes the target you have indicated and lets you choose another
one.
Configuring the descriptor
Options
The Options section can be expanded to reveal the following options:
- Auto-apply changes - When selected, auto-applies changes to target and anchor elements.
- Responsive websites - When selected, enables the anchor to automatically move from left to the top of the target, or from top to the left of the target, thus letting you automate websites that are responsive.
- Element visibility -
Checks whether the UI element is visible or not. There are three options
available from the drop-down menu:
- None - Does not check for visibility.
- Interactive (for Fuzzy selector) - Default option. Checks if the element is potentially visible, ignoring page scroll and obstructions by other apps, or the fact that the application is minimized. This check is useful when trying to ensure that you are not targeting invisible elements that exist in the DOM but are hidden.
- Fully visible - Ensures that the UI element is visible on the screen.
Window selector
The Window Selector section can be expanded to view and edit the selector of the indicated target application. You can also use the Open in UI Explorer button to edit the window selector in the UI Explorer.
Target
The under-the-hood process of finding an UI element uses several targeting methods:
- Strict selector
- Fuzzy selector
- Image
- Computer Vision
- The primary targeting method is represented by Selectors: Strict selector or Fuzzy selector.
- The secondary targeting method is Computer Vision.
- The tertiary targeting method is Image (disabled by default).
The icon
is displayed next to the leading targeting method at design time, simulating the
scenario at runtime.
Also, it's worth mentioning that the Wait for primary targeting method until timeout project setting (Project Settings > UI Automation Modern), which is enabled by default in new projects starting with activity package version 23.10.3 and which can be enabled manually for projects created prior to this activity package version, offers the primary targeting method some extra time to consume its full timeout until any fallback targeting method can be used.
If one of the primary targeting methods manages to find the UI element until the timeout expires, then it's going to be used as the method to interact with the UI element. Strict selector is the targeting method that first searches for the UI element, followed shortly by the Fuzzy selector. Also, Strict selector has priority over Fuzzy selector.
When the UI element is not found by the primary targeting methods until the timeout expires, so if after all the retries within the allocated timeout neither Strict selector nor Fuzzy selector find the UI element, only then the secondary and the tertiary targeting methods start searching for the UI element within a timeout of their own.
The Wait for primary targeting method until timeout project setting also brings more predictability and visibility into selector related issues so that you can have better control over your UI elements targeting, while also having the solid fallback that Computer Vision provides.
If you disable this project setting, then all the targeting methods compete in a single timeout to find the UI element according to the hierarchy: a target found by lower ranking targeting method is used only if a higher ranking targeting method fails to do so.
Another important project setting in this context is Show closest matches in the 'Element not found' error message (Project Settings > UI Automation Modern), which is as well enabled by default in new projects starting with activity package version 23.10.3 and which can be enabled manually for projects created prior to this activity package version. When this setting is enabled, if a selector fails, a warning is displayed in the logs showcasing the method used to find the UI element together with ten suggestions of the closest selector matches that can be used to replace the faulty selector.
It is not recommended to manually edit the descriptor, as this entire process is thought out to be seamless for the user and should provide good results without the need for advanced configuration. Still, if you want to configure more advanced features for your descriptor, you can enable, disable, and edit the targeting methods.
Strict selector
Selectors are XML fragments that contain the attributes of UI elements and their parents. Selectors are used for targeting UI elements on the screen, by inspecting them and their attributes, and matching them with the attributes in the selector. You can find out more on what a selector is in the UiPath ecosystem by viewing this page.
The Strict selector of the target element can be edited from its dedicated section in the Selection Options window. You can also use the Open in UI Explorer button to edit the selector in the UI Explorer.
The Strict selector field supports the use and creation of variables and arguments, from the context menu (accessible by right-clicking).
You can also select a part of the selector and use the context menu to directly convert it into a variable that is pre-populated with your selection.
You can also select an already created variable from the Variable Name drop-down menu and use it in your selector.
The Override default value check box enables you to replace the value in your predefined variable with your current selection.
The Confirm button saves the changes you have made in your selector.
The following hotkeys are available:
Ctrl
+Space
: choose variableCtrl
+Shift
+Space
: choose argumentCtrl
+K
: create variableCtrl
+M
: create argument
If there are both a variable and an argument with the same name, the variable is always defaulted to and used at runtime.
Arguments with names identical to variables are hidden from the variable selection menu in the Selection Screen.
Changes done with variables and arguments can be propagated throughout all the other instances where they are found. For example, if a variable is used in both the Strict selector and Fuzzy selector fields, changing it in one of the fields generates a message that informs you the same value was found in another field, and offering you the option to replace all instances with the new value.
Strict selectors also support wildcards, which can be used to replace zero or multiple characters in a string. You can find out more about this functionality here.
Fuzzy selector
The Fuzzy selector is very similar to Strict selector, with the difference that its attributes are fuzzy matched instead of perfectly matched. This means that a Fuzzy selector can output multiple elements, as opposed to Strict selector which outputs a single element at a time.
The Fuzzy selector can be edited from its dedicated section in the Selection Options window.
The matching accuracy of the Fuzzy selector is based on the Levenshtein distance algorithm to count the similarity of selectors. The default value is 0.5 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The matching accuracy can be set within a range of 0.4 to 1 where a lower value indicates a higher degree of fuzziness, meaning the Fuzzy selector outputs more elements, and a higher value indicates a lower degree of fuzziness, meaning the Fuzzy selector outputs less elements.
Just like the Strict selector, the Fuzy selector field supports the use and creation of variables and arguments and it also supports wildcards. For more details, refer to the information provided above.
If the indicated element contains text, the text is displayed in the input box below the Fuzzy selector edit window. If text is displayed in this input box, at runtime it must be found in the element itself to be considered a match.
The drop-down menu next to the input box has the following options:
- Text is/InnerText is/FullText is (depending on the text scraping method used) - displays the text that is scraped; the best scraping method is automatically identified and it is recommended to use it
- Ignore text - ignores the text specified in the input box
Image
If Image automation is used for your target, you can also view the screenshot of the element, as well as adjust the image accuracy. The default value is 0.8 and it can be configured either from the Accuracy slider or directly from the input field by typing a valid value. The accuracy can be set within a range of 0.4 to 1. For further information, check the Image and Text Automation section.
If you want to define Image-only targets, you can use the Image region selection (F3) or draw a box selection of the area you want to use as your target/anchor while indicating.
You can also use the refresh button to manually refresh the image that you have
selected, in case the image of the element changed since it was indicated.
The Image targeting method is disabled by default in Project Settings.
To enable it at project level, go to Project Settings > UI Automation Modern > Targeting methods - Desktop applications / Targeting methods - Web browsers / Targeting methods - Java / Targeting methods - SAP (depending on what type of application you want to automate) > Image > select True.
To enable the method on a specified element, the element must be valid.
Computer Vision
Native Text
If you selected the Enable Native text check box when indicating the element, the Native Text targeting method is used. This method enables you to use all the text labels that are identified with the Native method for identifying the element you want to interact with.
The text label identified through this method can be edited from its dedicated section, and you can specify whether you want the text to be case sensitive or not by using the Toggle case sensitive button.
Offline editing
Selecting the Edit offline option opens the selection screen in offline editing mode. In offline editing mode, you cannot add anchors, validate the descriptor, or other functionality that requires a live application.
Validating the descriptor
The validation process ensures that all combinations of search methods for the target are working correctly. You can validate your selection by using the Validate button in the Selection Screen.
If you have a target and anchor pair, each of them uses all the available methods for targeting. To find the final target, all possible combinations for each of these elements are matched to elements on the screen and a score is given after the match.
After performing the validation, if there are any duplicates found, they are highlighted in the target application. If no duplicates are found, icons appear alongside the targeting methods, informing you of the state of each targeting method in the given situation. The following symbols are used:
- the
targeting method that was fastest
- the method worked, but was not the fastest
- the method found duplicates and the correct element
was identified by using this method alongside other targeting methods
- the
method found duplicates and the correct element could not be identified by using
this method
- the
method failed to identify any elements