Computer Vision activities

This section includes Computer Vision related activities found in the UiPath.UIAutomation.Activities package.

• CV Check
• CV Click
• CV Dropdown Select
• CV Element Exists
• CV Extract Table
• CV Get Text
• CV Highlight
• CV Hover
• CV Refresh
• CV Screen Scope
• CV Type Into

Note: The Computer Vision activities are not compatible with Windows 7.

The Computer Vision activities contain refactored fundamental UI Automation activities such as Click, Type Into, or Get Text. The main difference between the Computer Vision activities and their classic counterparts is their usage of the Computer Vision neural network developed in-house by our Machine Learning department. The neural network is able to identify UI elements such as buttons, text input fields, or check boxes without the use of selectors.

Created mainly for automation in virtual desktop environments, such as Citrix machines, these activities bypass the issue of non-existent or unreliable selectors, as they send images of the window you are automating to the neural network, where it is analyzed and all UI elements are identified and labeled according to what they are. Smart anchors are used to pinpoint the exact location of the UI element you are interacting with, ensuring the action you intend to perform is successful.

Using the Computer Vision activities

All the Computer Vision activities function only when inside a CV Screen Scope activity, which establishes the actual connection to the neural network server, thus enabling you to analyze the UI of the applications you want to automate.

Any workflow using the Computer Vision activities must begin with dragging a CV Screen Scope activity to the designer panel.

Once this is done, the Indicate on screen button in the body of the scope activity can be used to select the area of the screen that you want to work in.

Note: Double-clicking the informative screenshot displays the image that has been captured and highlights in purple all of the UI elements that have been identified by the neural network and OCR engine.

Note: Area selection can also be used to indicate only a portion of the UI of the application you want to automate. This is especially useful in situations where there are multiple text fields that have the same label and cannot be properly identified.

Once a CV Screen Scope activity is properly configured, you can start using all of the other activities in the pack to build your automation.

Indicate on screen

The activities that perform actions on UI elements can be configured at design time by using the Indicate on screen button found in the activity card. The activities that have this feature are:

• CV Check
• CV Click
• CV Dropdown Select
• CV Element Exists
• CV Extract Table
• CV Get Text
• CV Highlight
• CV Hover
• CV Screen Scope
• CV Type Into

Clicking the Indicate on screen (hotkey: I) button opens the helper wizard.

Indicate

The Indicate field specifies what you are indicating at the moment. When the helper is opened for the first time, the Target needs to be indicated. For each possible target, the wizard automatically selects an anchor, if one is available.

Show elements

The Show Elements (hotkey: s) button in the wizard highlights all UI elements that have been identified by the Computer Vision analysis, making it easier for you to choose what to interact with.

Screen Refresh

The Screen Refresh (hotkey: F5) button can be used at design time, in case something changes in the target app, enabling you to send a new picture to the Computer Vision server to be analyzed again.

Delayed Screen Refresh

The Delayed Screen Refresh (hotkey: F2) button performs a refresh of the target app after waiting 3 seconds. The run-time auto-scroll direction is also captured (ScrollDirection property).

Relative To

The activities that have the Indicate wizard also include a Relative To feature, where the target is a stable element and you drag a distance relative to it. The main target, the stable one, acts as a proxy target for the second target, the relative one.

This is useful when you want to select an area that is not correspondent to the UI elements that the Computer Vision engine has identified, or when you want to select a general area in the UI without specifying a certain UI element.

The relative target can be either a single point or an area selection in the application you are automating.

Single point selection is normally placed in a spot where Computer Vision detection is not accurate and might not find certain elements. The activities that use single point selection are CV Click, CV Hover, and CV Type Into.

Area selection is used by CV Get Text and is automatically activated for this activity.

Note: Both a point or an area used as a target for Relative To are scalable with the resolution scale you have chosen for your screen. If the scale of the application itself is variable, instead, this feature loses its reliability.

As a general rule, any anchor that you add must be found at runtime, meaning that if at least one anchor is not found, the entire selection fails to be found. So, anchors need to be stable elements that you know for certain Computer Vision can detect every time and that do not change drastically in the UI at runtime.

Let's say you want to click the "X" button in the image below. You can notice that the target is on the "X" button, but the click happens in the green circle; this is because you have chosen to click relative to the "X" button. So, unless Computer Vision cannot detect a certain UI element, there is no need to use the Relative To feature.

To select the "X" button and not an area relative to it, make sure the "X" button has a green outline on it.

If the "X" button is not stable, you can click relative to a stable element.

Table indication

The Computer Vision activities also offer support for indicating tables. Targeting in tables can be done by selecting a cell you want to interact with, which prompts the neural network to automatically identify the column and the row that define the position of that cell, displaying them in a grid.

By default, the names of the column and row are used in the descriptor to pinpoint the location of the cell. Clicking the column and row indexes automatically adds them to your descriptor. This might be useful in situations where column and row names are changed, but you want to extract the same position of a cell.

After successfully indicating the Target, the wizard closes and the activity is configured with the target you selected.

Note: Please remember that whenever you choose to submit errors in the behavior of the neural network, you are helping it learn and indirectly helping us give you a better product. Submit as many issues as you can, as this gives us the opportunity to acknowledge and fix them.