- Visão geral
- Automação de Interface Gráfica
- Aplicativos e tecnologias automatizados com a Automação de Interface Gráfica
- Compatibilidade do projeto
- UI-ANA-016 - Extrair URL de navegador aberto
- UI-ANA-017 - ContinuarComErro verdadeiro
- UI-ANA-018 - Listar atividades de OCR/Imagem
- UI-DBP-006 - Uso do contêiner
- UI-DBP-013 - Uso Incorreto da Automação do Excel
- UI-DBP-030 - Uso de variáveis proibidas em seletores
- UI-PRR-001 - Simular clique
- UI-PRR-002 - Simular Digitação
- UI-PRR-003 - Uso incorreto do aplicativo aberto
- UI-PRR-004 - Atrasos Embutidos em Código
- UI-REL-001 - Idx grande em Seletores
- UI-SEC-004 - Dados de e-mail do seletor
- UI-SEC-010 — restrições de app/URL
- UI-USG-011 - Atributos não permitidos
- UX-SEC-010 — restrições de app/URL
- UX-DBP-029 - Uso de senha não segura
- UI-PST-001 - Nível de log de auditoria nas configurações do projeto
- Ferramenta de Migração de Navegador do UiPath
- Região do Recorte
- Computer Vision Recorder
- Ativar
- Anchor Base
- Anexar Navegador
- Anexar Janela
- Block User Input
- Texto Explicativo
- Marcar
- Click
- Clicar na Imagem
- Click Image Trigger
- Click OCR Text
- Clicar no Texto
- Click Trigger
- Close Application
- Fechar Guia
- Close Window
- Context Aware Anchor
- Copy Selected Text
- Element Attribute Change Trigger
- Element Exists
- Element Scope
- Element State Change Trigger
- Export UI Tree
- Extrair Dados Estruturados
- Find Children
- Localizar Elemento
- Localizar Imagem
- Find Image Matches
- Find OCR Text Position
- Localizar Elemento Relativo
- Find Text Position
- Get Active Window
- Get Ancestor
- Get Attribute
- Get Event Info
- Get From Clipboard
- Obter Texto Completo
- Get OCR Text
- Get Password
- Get Position
- Get Source Element
- Get Text
- Obter Texto Visível
- Voltar
- Avançar
- Ir para a Página Inicial
- Google Cloud Vision OCR
- Hide Window
- Highlight
- Hotkey Trigger
- Hover
- Focalizar Imagem
- Hover OCR Text
- Focalizar Texto
- Imagem Existe
- Indicate On Screen
- Inject .NET Code
- Inject Js Script
- Invoke ActiveX Method
- Key Press Trigger
- Carregar Imagem
- Maximize Window
- Microsoft Azure Computer Vision OCR
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Minimize Window
- Monitorar eventos
- Mouse Trigger
- Move Window
- Navegar Para
- OCR Text Exists
- On Element Appear
- On Element Vanish
- Aparecer Na Imagem
- Desaparecer Na Imagem
- Abrir Aplicativo
- Abrir Navegador
- Atualizar Navegador
- Replay User Event
- Restore Window
- Save Image
- Select Item
- Select Multiple Items
- Enviar Tecla de Acesso
- Set Clipping Region
- Set Focus
- Set Text
- Set To Clipboard
- Set Web Attribute
- Show Window
- Iniciar Processo
- System Trigger
- Take Screenshot
- Tesseract OCR
- Texto Existe
- Dica de ferramenta
- Type Into
- Digitar Texto Seguro
- Use Foreground
- Wait Attribute
- Esperar Elemento Desaparecer
- Aguardar Imagem Desaparecer
- Application event trigger
- Check/Uncheck
- Check App State
- Check Element
- Click
- Click Event Trigger
- Drag and Drop
- Extract Table Data
- For Each UI Element
- Get Attribute
- Get Browser Data
- Get Text
- Get URL
- Go To URL
- Highlight
- Hover
- Inject Js Script
- Keyboard Shortcuts
- Keypress Event Trigger
- Mouse Scroll
- Navigate Browser
- Select Item
- Set Browser Data
- Set Runtime Browser
- Set Text
- Take Screenshot
- Type Into
- Use Application/Browser
- Realize pesquisa no navegador e recupere resultados usando APIs de Automação de interface gráfica
- Navegação na Web
- Localizar imagens
- Clicar em imagens
- Disparar e monitorar eventos
- Criar e substituir arquivos
- Páginas HTML: extrair e manipular informações
- Manipulação de janelas
- Seleção automatizada de listas
- Localizar e manipular elementos de janela
- Gerenciar automação de texto
- Carregar e processar imagens
- Gerenciar ações ativadas pelo mouse
- Automatizar o runtime de aplicativos
- Execução automatizada de um aplicativo local
- Navegação em navegador
- Automação da Web
- Exemplo de escopo de disparador
- Enable UI Automation support in DevExpress
- Computer Vision Local Server
- Automação Móvel
- Notas de versão
- Compatibilidade do projeto
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Instalar Aplicativo
- Gerenciar Aplicativo Atual
- Manage Other App
- Open DeepLink
- Abrir URL
- Mobile Device Connection
- Deslizar Direcional
- Padrão de Desenho
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Obter parte da captura de tela
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Deslizar
- Tap
- Type Text
- Creating variables from selector attributes
- Introdução às APIs de automação móvel
- Using mobile automation for mobile banking application
- Terminal
- Notas de versão
- Sobre o pacote de atividades Terminal
- Compatibilidade do projeto
- Melhores práticas
- Find Text
- Get Color At Position
- Get Cursor Position
- Get Field
- Obter Campo na Posição
- Obter Área da Tela
- Get Text
- Obter Texto na Posição
- Mover Cursor
- Mover cursor para o texto
- Enviar Tecla de Controle
- Send Keys
- Send Keys Secure
- Set Field
- Definir Campo na Posição
- Terminal Session
- Aguardar Texto do Campo
- Aguardar Texto da Tela
- Aguardar Texto na Posição
Using mobile automation for mobile banking application
In this tutorial, we explore how to use the Mobile Automation package to automate tasks within a mobile banking application. The tutorial uses the mobile banking simulator provided by Bank of America, and a Google Pixel Emulator from Sauce Labs. We will specifically focus on automating and testing the following actions:
- Logging into the app.
- Implementing a debit card lock.
- Viewing a claim and capturing a screenshot.
The tutorial provides instructions on how to use both activities and APIs for automating these scenarios.
- Add a device from Sauce Labs into MDM.
For more information about adding devices, visit Adding Sauce Labs device.
- Add the Bank of America mobile banking simulator into MDM.
For more information about adding applications, visit Adding applications.
To follow the steps and try out the tutorial yourself, check the following sample project: Mobile Banking Simulator.
In this example we use Object Repository elements as input UI elements to activities. To create UI elements for mobile, establish a connection in Mobile Device Manager (MDM) between the target device and application. Then, directly in the Object Repository panel, create each element and use Indicate Element to indicate the specific UI element on the MDM device.
To easily follow the tutorial, check the following list for the Object Repository structure:
- BankOfAmerica application, MobileScreen menu
- backFromClaimDetails
- backFromClaimStatus
- BackToMenu
- firstClaimSeeDetails
- Lock
- LockCard
- Login
- ManageDebitCredit
- Menu
- Senha
- swipeMenu
- ID do Usuário
- viewmyclaims
- Establish a connection between the device and application:
- Add a Mobile Device Connection activity.
- Use Select Connection Details to connect the previously created mobile device and application.
- Login to the mobile application:
- Add a Tap activity to select the User ID field, so you can then enter a username.
- Add a Set Text activity to input the username inside the User ID field.
- Add a Tap activity to select the Password field, so you can then enter the password.
- Add a Set Text activity to input the corresponding password.
- Use a Tap activity to select the Login button.
Figure 1. The first activities as part of logging into the application
- Lock a debit or credit card:
- Add a Tap activity to navigate to the mobile banking app menu.
- Add a Swipe activity to swipe down in the menu.
The goal is to swipe until you find the Manage Debit/Credit section.
- Add a Tap activity to select the Manage Debit/Credit section.
- Add a Tap activity to select the Lock icon for the first card that is displayed.
- Add a Swipe activity to swipe down in the lock card notice.
This action allows you to confirm the action by selecting Lock Card.
- Add a Tap activity to select Lock Card.
Figure 2. Activities used for locking the debit card
- Take a screenshot of a claim:
- Add a Tap activity to select the back button, and return to the main menu.
- Add a Swipe activity to swipe down in the main menu, until the View My Claims section appears.
- Use a Tap activity to select the View My Claims section.
- Add a Tap activity to select See details for the first claim.
- Use the TakeScreenshot API to take a screenshot of the first claim details.
Figure 3. Activities used to take a screenshot of a claim
- Finally, after taking the screenshot, we go back to the main menu of the app, by using the Tap activity to select the Back UI element twice.
To automate a mobile application using APIs, record the application mobile elements within the project's Object Repository. For this example, we created a BankOfAmerica application with a general MobileScreen, containing all the UI elements that we use to automate. To create UI elements for mobile, establish a connection in Mobile Device Manager (MDM) between the target device and application. Then, directly in the Object Repository panel, create each element and use Indicate Element to indicate the specific UI element on the MDM device.
Check the following list for the structure of the Object Repository:
- BankOfAmerica application, MobileScreen menu
- backFromClaimDetails
- backFromClaimStatus
- BackToMenu
- firstClaimSeeDetails
- Lock
- LockCard
- Login
- ManageDebitCredit
- Menu
- Senha
- swipeMenu
- ID do Usuário
- viewmyclaims
- Establish a connection between the device and application:
- Create a variable of type
Connection
, using the Connect API, with the second overload version.For this overload, enter the device name and application name as Strings, to establish the connection.
[TestCase] public void Execute() { Connection connection = mobile.Connect( "Android mobile device", "BankOfAmericaSimulator"); Log("Connection established");
[TestCase] public void Execute() { Connection connection = mobile.Connect( "Android mobile device", "BankOfAmericaSimulator"); Log("Connection established");
- Create a variable of type
- Optionally, for debugging purposes, you can add a
Log
message after the connection. - Log in to the mobile application:
- Call the Tap API to click the User ID field.
This action allows you to enter the username.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId); - Use the SetText API to enter a username in the User ID
field.
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId, "john.doe@uipath.com", MobileOptions.SetText().WithSendNewline(false));
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId, "john.doe@uipath.com", MobileOptions.SetText().WithSendNewline(false)); - Call the Tap API to
select the Password field so you can enter the
password.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password); - Use the Set Text API to enter a password in the Password field.
The
WithSendNewLine(false)
method ensures that the OK keyboard element is not pressed on the mobile device after sending the text. For more information about the available options of the Set Text API, visit MobileOptions.SetTextOptions Class.connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password, "abcde", MobileOptions.SetText().WithSendNewline(false));
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password, "abcde", MobileOptions.SetText().WithSendNewline(false)); - Employ the Tap API to
select
Login.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Login);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Login);
- Call the Tap API to click the User ID field.
- Lock a debit or credit card:
- Navigate to the menu of the mobile banking app, using the Tap
API:
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Menu);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Menu); - Swipe down in the menu, using the Swipe API.
The goal is to swipe until you find the Manage Debit/Credit section.
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, SwipeDirection.Down, 4);
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, SwipeDirection.Down, 4); - Select the Manage Debit/Credit section using the Tap API.
The
WithTimeout()
method enables the API to pause for a predetermined number of seconds. This pause lasts until the target UI element appears, or an error is thrown.connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, TapTypeEnum.Single, MobileOptions.Tap().WithTimeout(120));
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, TapTypeEnum.Single, MobileOptions.Tap().WithTimeout(120)); - For the first card that
is displayed, select the Lock icon using the Tap
API.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Lock);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Lock); - Swipe down in the lock
card notice, using the Directional Swipe API, so we can confirm the action by
selecting Lock
Card.
connection.DirectionalSwipe(SwipeDirection.Down);
connection.DirectionalSwipe(SwipeDirection.Down); - Select Lock Card using the Tap
API.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.LockCard);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.LockCard);
- Navigate to the menu of the mobile banking app, using the Tap
API:
- Take a screenshot of a claim.
- Go back to the main menu,
using the Tap API and the back
icon.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.BackToMenu);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.BackToMenu); - Swipe down in the main menu, until the View My Claims section appears.
For this step, we use the Swipe API, this time employing the sixth overload. This overload requires an
IElementDescriptor
for the swipe container, and aSelectorTarget
for the target UI element.We first create aSelectorTarget
for the View My Claims section, and then employ the Swipe API.- Open MDM and create the connection between the mobile device and application.
- Go to the Object Repository and create a new element for the mobile screen.
- Select Indicate element.
- From the selection options menu, copy the web control identified by the selectors.
For this mobile banking app, you have to ensure that WebView selectors as native selectors is selected as a selection option.
- Return to the
code editor, and create a variable of type
SelectorTarget
. Use theMobileTarget.FromSelector
method which requires as argument the web control you previously copied.SelectorTarget viewMyClaimsTarget = MobileTarget.FromSelector("<mbl android:className='android.widget.TextView' text='View My Claims' />");
SelectorTarget viewMyClaimsTarget = MobileTarget.FromSelector("<mbl android:className='android.widget.TextView' text='View My Claims' />"); - Use the Swipe
API, with the
SelectorTarget
variable, to finish the swipe down action.connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, viewMyClaimsTarget, SwipeDirection.Down, 20);
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, viewMyClaimsTarget, SwipeDirection.Down, 20);
- Tap the View My
Claims
section.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.viewmyclaims);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.viewmyclaims); - Tap See details
for the first
claim.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.firstClaimSeeDetails);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.firstClaimSeeDetails); - Use the TakeScreenshot API to take a screenshot of the first claim
details.
connection.TakeScreenshot();
connection.TakeScreenshot();
- Go back to the main menu,
using the Tap API and the back
icon.
- Finally, after taking the screenshot, we go back to the main menu of the app, by
tapping the Back UI element
twice.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimDetails); connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimStatus);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimDetails); connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimStatus);