The AI Computer Vision pack contains refactored fundamental UIAutomation activities such as Click, Type Into, or Get Text. The main difference between the CV activities and their classic counterparts is their usage of the Computer Vision neural network developed in-house by our Machine Learning department. The neural network is able to identify UI elements such as buttons, text input fields, or check boxes without the use of selectors.
Created mainly for automation in virtual desktop environments, such as Citrix machines, these activities bypass the issue of inexistent or unreliable selectors, as they send images of the window you are automating to the neural network, where it is analyzed and all UI elements are identified and labeled according to what they are. Smart anchors are used to pinpoint the exact location of the UI element you are interacting with, ensuring the action you intend to perform is successful.
Due to possible internal dependency issues, it is highly recommended to remove the default UIAutomation activity package that each new process comes with by default before installing the Computer Vision activities pack.
To install the Computer Vision pack to a new project, browse for it in the Package Manager.
After the activity package is installed in the current project, you need an ApiKey which can be obtained from the Cloud Platform as detailed here
The following table lists the compatibility between versions of the Computer Vision and the UIAutomation activity packages.
All of the activities in this pack only function when inside a CV Screen Scope activity, which establishes the actual connection to the neural network server, thus enabling you to analyze the UI of the apps you want to automate. Any workflow using the Computer Vision activities must begin with dragging a CV Screen Scope activity to the Designer panel. Once this is done, the Indicate on screen button in the body of the scope activity can be used to select the area of the screen that you want to work in.
Double-clicking the informative screenshot displays the image that has been captured and highlights in purple all of the UI elements that have been identified by the neural network and OCR engine.
Area selection can also be used to indicate only a portion of the UI of the application you want to automate. This is especially useful in situations where there are multiple text fields that have the same label and cannot be properly identified.
Once a CV Screen Scope activity is properly configured, you can start using all of the other activities in the pack to build your automation.
The activities that perform actions on UI elements can be configured at design time by using the Indicate On Screen button present in the body of the activities. The activities that have this feature are:
Clicking the Indicate On Screen (hotkey: I) button opens the helper wizard.
The CV Click, CV Hover, and CV Type Into activities also feature a Relative To button in the helper wizard, which enables you to configure the target as being relative to an element.
The Indicate field specifies what you are indicating at the moment. When the helper is opened for the first time, the Target needs to be indicated. For each possible target, the wizard automatically selects an anchor, if one is available.
After successfully indicating the Target, the wizard closes and the activity is configured with the target you selected.
If no unique anchor is automatically identified, the Indicate field informs you of this fact, enabling you to indicate additional Anchors, which make the target easier to find.
The Show Elements (hotkey: s) button in the wizard highlights all UI elements that have been identified by the Computer Vision analysis, making it easier for you to choose what to interact with.
The Refresh Scope (hotkey: F5) button can be used at design time, in case something changes in the target app, enabling you to send a new picture to the CV server to be analyzed again.
The Refresh After Delay (hotkey: F2) button performs a refresh of the target app after waiting 3 seconds.
Please remember that whenever you choose to submit errors in the behavior of the neural network, you are helping it learn and indirectly helping us give you a better product. Submit as many issues as you can, as this gives us the opportunity to acknowledge and fix them.