- Release notes
- UI-ANA-016 - Pull Open Browser URL
- UI-ANA-017 - ContinueOnError True
- UI-ANA-018 - List OCR/Image Activities
- UI-DBP-006 - Container Usage
- UI-DBP-013 - Excel Automation Misuse
- UI-DBP-030 - Forbidden Variables Usage In Selectors
- ST-DBP-021 - Hardcoded Timeout
- UI-PRR-001 - Simulate Click
- UI-PRR-002 - Simulate Type
- UI-PRR-003 - Open Application Misuse
- UI-PRR-004 - Hardcoded Delays
- UI-REL-001 - Large Idx in Selectors
- UI-SEC-004 - Selector Email Data
- UI-SEC-010 - App/Url Restrictions
- UI-USG-011 - Non Allowed Attributes
- SY-USG-013 - Private File Invoke
- SY-USG-014 - Incorrect Execution Template Placeholders
- Applications and technologies automated with UI Automation
- UiPath Browser Migration Tool
- Computer Vision Recorder
- Click
- Hover
- Type Secure Text
- Send Hotkey
- Type Into
- Check
- Select Item
- Select Multiple Items
- Get Text
- Set Text
- Activate
- Highlight
- Set Focus
- Anchor Base
- Context Aware Anchor
- Find Relative Element
- Indicate On Screen
- Element Exists
- Find Element
- Wait Element Vanish
- Find Children
- Get Ancestor
- Element Scope
- Get Attribute
- Set Clipping Region
- Get Position
- Take Screenshot
- Wait Attribute
- On Element Appear
- On Element Vanish
- Click Text
- Hover Text
- Find Text Position
- Get Full Text
- Get Visible Text
- Extract Structured Data
- Text Exists
- Click OCR Text
- Hover OCR Text
- Get OCR Text
- Find OCR Text Position
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Microsoft Azure Computer Vision OCR
- Tesseract OCR
- Google Cloud Vision OCR
- OCR Text Exists
- Click Image
- Hover Image
- Find Image Matches
- Image Exists
- Find Image
- Wait Image Vanish
- On Image Appear
- On Image Vanish
- Load Image
- Save Image
- Attach Browser
- Close Tab
- Go Back
- Go Forward
- Go Home
- Inject JS Script
- Navigate To
- Open Browser
- Refresh Browser
- Set Web Attribute
- Attach Window
- Close Window
- Get Active Window
- Hide Window
- Maximize Window
- Minimize Window
- Move Window
- Restore Window
- Show Window
- Export UI Tree
- Invoke ActiveX Method
- Inject .NET Code
- Click Trigger
- Key Press Trigger
- Click Image Trigger
- System Trigger
- Hotkey Trigger
- Mouse Trigger
- Get Event Info
- Monitor Events
- Get Source Element
- Replay User Event
- Block User Input
- Use Foreground
- Element Attribute Change Trigger
- Element State Change Trigger
- Callout
- Copy Selected Text
- Get From Clipboard
- Set To Clipboard
- Close Application
- Open Application
- Start Process
- Get Password
- Tooltip
- Click Picture on Screen
- Click Toolbar Button
- Read Statusbar
- Select Dates in Calendar
- Select Menu Item
- Expand Tree
- Call Transaction
- SAP Logon
- SAP Login
- Table Cell Scope
- SAP Session Attributes Change Trigger
- Expand ALV Tree
- Expand ALV Hierarchical Table
- Use Application/Browser
- Check App State
- Check/Uncheck
- Click
- Extract Table Data
- Get Text
- Get URL
- Go to URL
- Highlight
- Hover
- Keyboard Shortcuts
- Navigate Browser
- Select Item
- Take Screenshot
- Type Into
- Get Attribute
- Get Browser Data
- Set Browser Data
- Mouse Scroll
- Inject Js Script
- Drag and Drop
- For Each UI Element
- Application Event Trigger
- Click Event Trigger
- Keypress Event Trigger
- Set Text
- Check Element
- Using the Computer Vision activities
- CV Screen Scope
- CV Click
- CV Element Exists
- CV Get Text
- CV Highlight
- CV Hover
- CV Type Into
- CV Refresh
- CV Dropdown Select
- CV Check
- CV Extract Table
- Perform browser search and retrieve results using UI Automation APIs
- Web Browsing
- Find Images
- Click Images
- Trigger and Monitor Events
- Create and Override Files
- HTML Pages: Extract and Manipulate Information
- Window Manipulation
- Automated List Selection
- Find and Manipulate Window Elements
- Manage Text Automation
- Load and Process Images
- Manage Mouse Activated Actions
- Automate Application Runtime
- Automated Run of a Local Application
- Browser Navigation
- Web Automation
- Trigger Scope Example
- Release Notes
- Project Compatibility
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Install App
- Manage Current App
- Manage Other App
- Open DeepLink
- Open URL
- Mobile Device Connection
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Take Screenshot Part
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Swipe
- Tap
- Type Text
- Draw Pattern
Using the Computer Vision activities
All the Computer Vision activities function only when inside a CV Screen Scope activity, which establishes the actual connection to the neural network server, thus enabling you to analyze the UI of the applications you want to automate.
Any workflow using the Computer Vision activities must begin with dragging a CV Screen Scope activity to the designer panel.
Once this is done, the Indicate on screen button in the body of the scope activity can be used to select the area of the screen that you want to work in.
Once a CV Screen Scope activity is properly configured, you can start using all of the other activities in the pack to build your automation.
Indicating On Screen
The activities that perform actions on UI elements can be configured at design time by using the Indicate On Screen button present in the body of the activities. The activities that have this feature are:
- CV Screen Scope
- CV Dropdown Select
- CV Check
- CV Extract Table
- CV Click
- CV Element Exists
- CV Get Text
- CV Highlight
- CV Hover
- CV Type Into
Clicking the Indicate On Screen (hotkey: I) button opens the helper wizard.
Indicate
The Indicate field specifies what you are indicating at the moment. When the helper is opened for the first time, the Target needs to be indicated. For each possible target, the wizard automatically selects an anchor, if one is available.
Show elements
The Show Elements (hotkey: s) button in the wizard highlights all UI elements that have been identified by the Computer Vision analysis, making it easier for you to choose what to interact with.
Screen Refresh
The Screen Refresh (hotkey: F5) button can be used at design time, in case something changes in the target app, enabling you to send a new picture to the Computer Vision server to be analyzed again.
Delayed Screen Refresh
The Delayed Screen Refresh (hotkey: F2) button performs a refresh of the target app after waiting 3 seconds. The run-time auto-scroll direction is also captured (ScrollDirection property).
Relative To
The activities that have the Indicate wizard also include a Relative To feature, where the target is a stable element and you drag a distance relative to it. The main target, the stable one, acts as a proxy target for the second target, the relative one.
This is useful when you want to select an area that is not correspondent to the UI elements that the Computer Vision engine has identified, or when you want to select a general area in the UI without specifying a certain UI element.
The relative target can be either a single point or an area selection in the application you are automating.
Single point selection is normally placed in a spot where Computer Vision detection is not accurate and might not find certain elements. The activities that use single point selection are CV Click, CV Hover, and CV Type Into.
Area selection is used by CV Get Text and is automatically activated for this activity.
As a general rule, any anchor that you add must be found at runtime, meaning that if at least one anchor is not found, the entire selection fails to be found. So, anchors need to be stable elements that you know for certain Computer Vision can detect every time and that do not change drastically in the UI at runtime.
Let's say you want to click the "X" button in the image below. You can notice that the target is on the "X" button, but the click happens in the green circle; this is because you have chosen to click relative to the "X" button. So, unless Computer Vision cannot detect a certain UI element, there is no need to use the Relative To feature.
To select the "X" button and not an area relative to it, make sure the "X" button has a green outline on it.
If the "X" button is not stable, you can click relative to a stable element.
Table Indication
The Computer Vision activities also offer support for indicating tables. Targeting in tables can be done by selecting a cell you want to interact with, which prompts the neural network to automatically identify the column and the row that define the position of that cell, displaying them in a grid.
By default, the names of the column and row are used in the descriptor to pinpoint the location of the cell. Clicking the column and row indexes automatically adds them to your descriptor. This might be useful in situations where column and row names are changed, but you want to extract the same position of a cell.
After successfully indicating the Target, the wizard closes and the activity is configured with the target you selected.