- Overview
- UI Automation
- Applications and technologies automated with UI Automation
- Project compatibility
- UI-ANA-016 - Pull Open Browser URL
- UI-ANA-017 - ContinueOnError True
- UI-ANA-018 - List OCR/Image Activities
- UI-DBP-006 - Container Usage
- UI-DBP-013 - Excel Automation Misuse
- UI-DBP-030 - Forbidden Variables Usage In Selectors
- UI-PRR-001 - Simulate Click
- UI-PRR-002 - Simulate Type
- UI-PRR-003 - Open Application Misuse
- UI-PRR-004 - Hardcoded Delays
- UI-REL-001 - Large Idx in Selectors
- UI-SEC-004 - Selector Email Data
- UI-SEC-010 - App/Url Restrictions
- UI-USG-011 - Non Allowed Attributes
- UX-SEC-010 - App/Url Restrictions
- UX-DBP-029 - Insecure Password Use
- UI-PST-001 - Audit Log Level in Project Settings
- UiPath Browser Migration Tool
- Clipping region
- Computer Vision Recorder
- Activities index
- Activate
- Anchor Base
- Attach Browser
- Attach Window
- Block User Input
- Callout
- Check
- Click
- Click Image
- Click Image Trigger
- Click OCR Text
- Click Text
- Click Trigger
- Close Application
- Close Tab
- Close Window
- Context Aware Anchor
- Copy Selected Text
- Element Attribute Change Trigger
- Element Exists
- Element Scope
- Element State Change Trigger
- Export UI Tree
- Extract Structured Data
- Find Children
- Find Element
- Find Image
- Find Image Matches
- Find OCR Text Position
- Find Relative Element
- Find Text Position
- Get Active Window
- Get Ancestor
- Get Attribute
- Get Event Info
- Get From Clipboard
- Get Full Text
- Get OCR Text
- Get Password
- Get Position
- Get Source Element
- Get Text
- Get Visible Text
- Go Back
- Go Forward
- Go Home
- Google Cloud Vision OCR
- Hide Window
- Highlight
- Hotkey Trigger
- Hover
- Hover Image
- Hover OCR Text
- Hover Text
- Image Exists
- Indicate On Screen
- Inject .NET Code
- Inject Js Script
- Invoke ActiveX Method
- Key Press Trigger
- Load Image
- Maximize Window
- Microsoft Azure Computer Vision OCR
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Minimize Window
- Monitor Events
- Mouse Trigger
- Move Window
- Navigate To
- OCR Text Exists
- On Element Appear
- On Element Vanish
- On Image Appear
- On Image Vanish
- Open Application
- Open Browser
- Refresh Browser
- Replay User Event
- Restore Window
- Save Image
- Select Item
- Select Multiple Items
- Send Hotkey
- Set Clipping Region
- Set Focus
- Set Text
- Set To Clipboard
- Set Web Attribute
- Show Window
- Start Process
- System Trigger
- Take Screenshot
- Tesseract OCR
- Text Exists
- Tooltip
- Type Into
- Type Secure Text
- Use Foreground
- Wait Attribute
- Wait Element Vanish
- Wait Image Vanish
- Application Event Trigger
- Check/Uncheck
- Check App State
- Check Element
- Click
- Click Event Trigger
- Drag and Drop
- Extract Table Data
- For Each UI Element
- Get Attribute
- Get Browser Data
- Get Text
- Get URL
- Go to URL
- Highlight
- Hover
- Inject Js Script
- Keyboard Shortcuts
- Keypress Event Trigger
- Mouse Scroll
- Navigate Browser
- Select Item
- Set Browser Data
- Set Runtime Browser
- Set Text
- Take Screenshot
- Type Into
- Use Application/Browser
- Perform browser search and retrieve results using UI Automation APIs
- Web Browsing
- Find Images
- Click Images
- Trigger and Monitor Events
- Create and Override Files
- HTML Pages: Extract and Manipulate Information
- Window Manipulation
- Automated List Selection
- Find and Manipulate Window Elements
- Manage Text Automation
- Load and Process Images
- Manage Mouse Activated Actions
- Automate Application Runtime
- Automated Run of a Local Application
- Browser Navigation
- Web Automation
- Trigger Scope Example
- Enable UI Automation support in DevExpress
- Computer Vision Local Server
- Mobile Automation
- Release notes
- Project compatibility
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Install App
- Manage Current App
- Manage Other App
- Open DeepLink
- Open URL
- Mobile Device Connection
- Directional Swipe
- Draw Pattern
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Take Screenshot Part
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Swipe
- Tap
- Type Text
- Creating variables from selector attributes
- Getting started with Mobile Automation APIs
- Using mobile automation for mobile banking application
- Terminal
UI Automation Activities
Using mobile automation for mobile banking application
In this tutorial, we explore how to use the Mobile Automation package to automate tasks within a mobile banking application. The tutorial uses the mobile banking simulator provided by Bank of America, and a Google Pixel Emulator from Sauce Labs. We will specifically focus on automating and testing the following actions:
- Logging into the app.
- Implementing a debit card lock.
- Viewing a claim and capturing a screenshot.
The tutorial provides instructions on how to use both activities and APIs for automating these scenarios.
- Add a device from Sauce Labs
into MDM.
For more information about adding devices, visit Adding Sauce Labs device.
- Add the Bank of America mobile banking
simulator into MDM.
For more information about adding applications, visit Adding applications.
To follow the steps and try out the tutorial yourself, check the following sample project: Mobile Banking Simulator.
In this example we use Object Repository elements as input UI elements to activities. To create UI elements for mobile, establish a connection in Mobile Device Manager (MDM) between the target device and application. Then, directly in the Object Repository panel, create each element and use Indicate Element to indicate the specific UI element on the MDM device.
To easily follow the tutorial, check the following list for the Object Repository structure:
- BankOfAmerica application,
MobileScreen menu
- backFromClaimDetails
- backFromClaimStatus
- BackToMenu
- firstClaimSeeDetails
- Lock
- LockCard
- Login
- ManageDebitCredit
- Menu
- Password
- swipeMenu
- UserId
- viewmyclaims
- Establish a connection between the device and application:
- Add a Mobile Device Connection activity.
- Use Select Connection Details to connect the previously created mobile device and application.
- Login to the mobile application:
- Add a Tap activity to select the User ID field, so you can then enter a username.
- Add a Set Text activity to input the username inside the User ID field.
- Add a Tap activity to select the Password field, so you can then enter the password.
- Add a Set Text activity to input the corresponding password.
- Use a Tap activity to select the Login button.
Figure 1. The first activities as part of logging into the application
- Lock a debit or credit card:
- Add a Tap activity to navigate to the mobile banking app menu.
- Add a Swipe activity to swipe down in the menu.
The goal is to swipe until you find the Manage Debit/Credit section.
- Add a Tap activity to select the Manage Debit/Credit section.
- Add a Tap activity to select the Lock icon for the first card that is displayed.
- Add a Swipe activity to swipe down in the lock card notice.
This action allows you to confirm the action by selecting Lock Card.
- Add a Tap activity to select Lock Card.
Figure 2. Activities used for locking the debit card
- Take a screenshot of a claim:
- Add a Tap activity to select the back button, and return to the main menu.
- Add a Swipe activity to swipe down in the main menu, until the View My Claims section appears.
- Use a Tap activity to select the View My Claims section.
- Add a Tap activity to select See details for the first claim.
- Use the TakeScreenshot API to take a screenshot of the first claim details.
Figure 3. Activities used to take a screenshot of a claim
- Finally, after taking the screenshot, we go back to the main menu of the app, by using the Tap activity to select the Back UI element twice.
To automate a mobile application using APIs, record the application mobile elements within the project's Object Repository. For this example, we created a BankOfAmerica application with a general MobileScreen, containing all the UI elements that we use to automate. To create UI elements for mobile, establish a connection in Mobile Device Manager (MDM) between the target device and application. Then, directly in the Object Repository panel, create each element and use Indicate Element to indicate the specific UI element on the MDM device.
Check the following list for the structure of the Object Repository:
- BankOfAmerica application,
MobileScreen menu
- backFromClaimDetails
- backFromClaimStatus
- BackToMenu
- firstClaimSeeDetails
- Lock
- LockCard
- Login
- ManageDebitCredit
- Menu
- Password
- swipeMenu
- UserId
- viewmyclaims
- Establish a connection between
the device and application:
- Create a variable of type
Connection
, using the Connect API, with the second overload version.For this overload, enter the device name and application name as Strings, to establish the connection.
[TestCase] public void Execute() { Connection connection = mobile.Connect( "Android mobile device", "BankOfAmericaSimulator"); Log("Connection established");
[TestCase] public void Execute() { Connection connection = mobile.Connect( "Android mobile device", "BankOfAmericaSimulator"); Log("Connection established");
- Create a variable of type
- Optionally, for debugging purposes, you can add a
Log
message after the connection. - Log in to the mobile
application:
- Call the Tap API to click the User ID field.
This action allows you to enter the username.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId); - Use the SetText API to enter a username in the User ID
field.
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId, "john.doe@uipath.com", MobileOptions.SetText().WithSendNewline(false));
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId, "john.doe@uipath.com", MobileOptions.SetText().WithSendNewline(false)); - Call the Tap API to
select the Password field so you can enter the
password.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password); - Use the Set Text API to
enter a password in the Password field.
The
WithSendNewLine(false)
method ensures that the OK keyboard element is not pressed on the mobile device after sending the text. For more information about the available options of the Set Text API, visit MobileOptions.SetTextOptions Class.connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password, "abcde", MobileOptions.SetText().WithSendNewline(false));
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password, "abcde", MobileOptions.SetText().WithSendNewline(false)); - Employ the Tap API to
select
Login.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Login);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Login);
- Call the Tap API to click the User ID field.
- Lock a debit or credit card:
- Navigate to the menu of the mobile banking app, using the Tap
API:
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Menu);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Menu); - Swipe down in the menu, using the Swipe API.
The goal is to swipe until you find the Manage Debit/Credit section.
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, SwipeDirection.Down, 4);
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, SwipeDirection.Down, 4); - Select the Manage
Debit/Credit section using the Tap API.
The
WithTimeout()
method enables the API to pause for a predetermined number of seconds. This pause lasts until the target UI element appears, or an error is thrown.connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, TapTypeEnum.Single, MobileOptions.Tap().WithTimeout(120));
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, TapTypeEnum.Single, MobileOptions.Tap().WithTimeout(120)); - For the first card that
is displayed, select the Lock icon using the Tap
API.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Lock);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Lock); - Swipe down in the lock
card notice, using the Directional Swipe API, so we can confirm the action by
selecting Lock
Card.
connection.DirectionalSwipe(SwipeDirection.Down);
connection.DirectionalSwipe(SwipeDirection.Down); - Select Lock Card using the Tap
API.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.LockCard);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.LockCard);
- Navigate to the menu of the mobile banking app, using the Tap
API:
- Take a screenshot of a claim.
- Go back to the main menu,
using the Tap API and the back
icon.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.BackToMenu);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.BackToMenu); - Swipe down in the main
menu, until the View My Claims section appears.
For this step, we use the Swipe API, this time employing the sixth overload. This overload requires an
IElementDescriptor
for the swipe container, and aSelectorTarget
for the target UI element.We first create aSelectorTarget
for the View My Claims section, and then employ the Swipe API.- Open MDM and create the connection between the mobile device and application.
- Go to the Object Repository and create a new element for the mobile screen.
- Select Indicate element.
- From the selection options menu, copy the web
control identified by the selectors.
For this mobile banking app, you have to ensure that WebView selectors as native selectors is selected as a selection option.
- Return to the
code editor, and create a variable of type
SelectorTarget
. Use theMobileTarget.FromSelector
method which requires as argument the web control you previously copied.SelectorTarget viewMyClaimsTarget = MobileTarget.FromSelector("<mbl android:className='android.widget.TextView' text='View My Claims' />");
SelectorTarget viewMyClaimsTarget = MobileTarget.FromSelector("<mbl android:className='android.widget.TextView' text='View My Claims' />"); - Use the Swipe
API, with the
SelectorTarget
variable, to finish the swipe down action.connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, viewMyClaimsTarget, SwipeDirection.Down, 20);
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, viewMyClaimsTarget, SwipeDirection.Down, 20);
- Tap the View My
Claims
section.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.viewmyclaims);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.viewmyclaims); - Tap See details
for the first
claim.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.firstClaimSeeDetails);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.firstClaimSeeDetails); - Use the TakeScreenshot API to take a screenshot of the first claim
details.
connection.TakeScreenshot();
connection.TakeScreenshot();
- Go back to the main menu,
using the Tap API and the back
icon.
- Finally, after taking the screenshot, we go back to the main menu of the app, by
tapping the Back UI element
twice.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimDetails); connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimStatus);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimDetails); connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimStatus);