- Vue d'ensemble (Overview)
- Automatisation de l'interface utilisateur
- Applications et technologies automatisées avec UI Automation
- Compatibilité du projet
- UI-ANA-016 - Extraire l'URL ouverte du navigateur
- UI-ANA-017 - ContinuerSurErreur (ContinueOnError) True
- UI-ANA-018 - Répertorier les activités d'OCR/d'image
- UI-DBP-006 - Utilisation du conteneur
- UI-DBP-013 - Utilisation abusive de l’automatisation Excel
- UI-DBP-030 - Utilisation de variables interdites dans les sélecteurs
- UI-PRR-001 - Simuler un clic
- UI-PRR-002 - Type de simulation
- UI-PRR-003 - Ouverture d'une utilisation abusive de l'application
- UI-PRR-004 - Délais codés en dur
- UI-REL-001 - Idx volumineux dans les sélecteurs
- UI-SEC-004 - Données d’e-mail du sélecteur
- UI-SEC-010 - Restrictions d'applications/d'URL
- UI-USG-011 - Attributs non autorisés
- UX-SEC-010 - Restrictions d'applications/d'URL
- UX-DBP-029 - Utilisation d'un mot de passe non sécurisé
- UI-PST-001 - Niveau du journal d'audit dans les paramètres du projet
- Outil de migration de navigateur UiPath
- Zone de détourage
- Enregistreur de Computer Vision
- Activer (Activate)
- Base d'ancrage (Anchor Base)
- Lier à un navigateur (Attach Browser)
- Lier à une fenêtre (Attach Window)
- Block User Input
- Légende (Callout)
- Vérifier (Check)
- Cliquer (Click)
- Cliquer sur l'image (Click Image)
- Déclencheur de clic image (Click Image Trigger)
- Cliquer sur le texte OCR (Click OCR Text)
- Cliquer sur le texte (Click Text)
- Déclencheur de clic (Click Trigger)
- Fermer l'application (Close Application)
- Fermer l'onglet (Close Tab)
- Fermer la fenêtre (Close Window)
- Context Aware Anchor
- Copier le texte sélectionné (Copy Selected Text)
- Element Attribute Change Trigger
- Élément existant (Element Exists)
- Étendue de l'élément (Element Scope)
- Element State Change Trigger
- Export UI Tree
- Extraire les données structurées (Extract Structured Data)
- Rechercher les enfants (Find Children)
- Rechercher l'élément (Find Element)
- Rechercher l'image (Find Image)
- Rechercher les correspondances de l'image (Find Image Matches)
- Rechercher une position de texte OCR (Find OCR Text Position)
- Rechercher l'élément relatif (Find Relative Element)
- Rechercher la position du texte (Find Text Position)
- Obtenir la fenêtre active (Get Active Window)
- Obtenir l'ancêtre (Get Ancestor)
- Obtenir l'attribut (Get Attribute)
- Obtenir les infos de l'événement (Get Event Info)
- Récupérer du presse-papiers (Get From Clipboard)
- Obtenir le texte complet (Get Full Text)
- Obtenir le texte OCR (Get OCR Text)
- Récupérer le mot de passe (Get Password)
- Obtenir la position (Get Position)
- Obtenir l'élément source (Get Source Element)
- Obtenir le texte (Get Text)
- Obtenir le texte visible (Get Visible Text)
- Revenir en arrière (Go Back)
- Avancer (Go Forward)
- Accéder à l'accueil (Go Home)
- Google Cloud Vision OCR
- Masquer la fenêtre (Hide Window)
- Mettre en surbrillance (Highlight)
- Déclencheur de raccourci (Hotkey Trigger)
- Pointer (Hover)
- Pointer sur l'image (Hover Image)
- Pointer sur le texte OCR (Hover OCR Text)
- Pointer sur le texte (Hover Text)
- Image existante (Image Exists)
- Indiquer sur l'écran (Indicate On Screen)
- Injecter du code .NET
- Inject Js Script
- Invoquer la méthode ActiveX
- Déclencheur de pression de touche (Key Press Trigger)
- Charger l'image (Load Image)
- Agrandir la fenêtre (Maximize Window)
- Microsoft Azure ComputerVision OCR
- Reconnaissance optique des caractères Microsoft (Microsoft OCR)
- Microsoft Project Oxford Online OCR
- Réduire la fenêtre (Minimize Window)
- Surveiller les événements (Monitor Events)
- Déclencheur de souris (Mouse Trigger)
- Déplacer la fenêtre (Move Window)
- Accéder à (Navigate To)
- Texte OCR existant (OCR Text Exists)
- Sur affichage de l'élément (On Element Appear)
- Sur disparition de l'élément (On Element Vanish)
- Sur apparition de l'image (On Image Appear)
- Sur disparition de l'image (On Image Vanish)
- Ouvrir l'application (Open Application)
- Ouvrir le navigateur (Open Browser)
- Actualiser le navigateur (Refresh Browser)
- Relire l'événement utilisateur (Replay User Event)
- Restaurer la fenêtre (Restore Window)
- Enregistrer l'image (Save Image)
- Sélectionner l'élément (Select Item)
- Sélectionner plusieurs éléments (Select Multiple Items)
- Envoyer le raccourci (Send Hotkey)
- Définir la zone de détourage (Set Clipping Region)
- Définir le focus (Set Focus)
- Définir le texte (Set Text)
- Placer dans le presse-papiers (Set To Clipboard)
- Définir l'attribut Web (Set Web Attribute)
- Afficher la fenêtre (Show Window)
- Déclencher le processus (Start Process)
- Déclencheur système (System Trigger)
- Prendre une capture d'écran (Take Screenshot)
- Tesseract OCR
- Texte existant (Text Exists)
- Info-bulle
- Saisir dans (Type Into)
- Saisir un texte sécurisé (Type Secure Text)
- Utiliser le premier plan
- Attendre un attribut (Wait Attribute)
- Attendre que l'élément disparaisse (Wait Element Vanish)
- Attendre que l'image disparaisse (Wait Image Vanish)
- Application event trigger
- Check/Uncheck
- Check App State
- Check Element
- Cliquer (Click)
- Click Event Trigger
- Glisser et déposer
- Extract Table Data
- For Each UiElement
- Obtenir l'attribut (Get Attribute)
- Get Browser Data
- Obtenir le texte (Get Text)
- Get URL
- Go To URL
- Mettre en surbrillance (Highlight)
- Pointer (Hover)
- Inject Js Script
- Raccourcis clavier
- Keypress Event Trigger
- Mouse scroll
- Navigate Browser
- Sélectionner l'élément (Select Item)
- Set Browser Data
- Définir le navigateur du runtime (Set Runtime Browser)
- Définir le texte (Set Text)
- Prendre une capture d'écran (Take Screenshot)
- Saisir dans (Type Into)
- Use Application/Browser
- Joindre
- Vérifier (Check)
- Cliquer (Click)
- Glisser et déposer
- Extraire des données
- Obtenir l'attribut (Get Attribute)
- ObtenirEnfants
- ObtenirCibleRuntime
- GetText
- Get URL
- GoToUrl
- Mettre en surbrillance (Highlight)
- Pointer (Hover)
- IsEnabled
- Raccourci clavier
- Mouse scroll
- Ouvrir
- Sélectionner l'élément (Select Item)
- Prendre une capture d'écran (Take Screenshot)
- Saisir dans (Type Into)
- ÉtatAttente
- Effectuez une recherche par navigateur et récupérez les résultats à l'aide des API UI Automation
- Navigation sur le Web
- Rechercher des images
- Cliquer sur des images
- Déclencher et surveiller des événements
- Créer et remplacer des fichiers
- Pages HTML : extraire et manipuler des informations
- Manipulation des fenêtres
- Sélection de liste automatisée
- Rechercher et manipuler des éléments de fenêtre
- Gérer l'automatisation du texte
- Charger et traiter des images
- Gérer les actions activées par la souris
- Automatiser l'exécution des applications
- Exécution automatisée d'une application locale
- Navigation avec le navigateur
- Automatisation Web
- Exemple de fonctionnalités du déclencheur
- Enable UI Automation support in DevExpress
- Computer Vision Local Server
- Automatisation mobile
- Notes de publication
- Compatibilité du projet
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Installer l'application
- Gérer l'application actuelle
- Gérer une autre application
- Ouvrir DeepLink
- Ouvrir l'URL
- Mobile Device Connection
- Balayer directionnel
- Dessiner un modèle
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Prendre une capture d'écran (Take Screenshot)
- Prendre une partie de capture d'écran
- Élément existant (Element Exists)
- Execute Command
- Obtenir l'attribut (Get Attribute)
- Get Selected Item
- Obtenir le texte (Get Text)
- Set Selected Item
- Définir le texte (Set Text)
- Balayer
- Tap
- Saisir texte
- Creating variables from selector attributes
- Premiers pas avec les API d’automatisation mobile
- Using mobile automation for mobile banking application
- Terminal
- Notes de publication
- À propos du package d'activités Terminal
- Compatibilité du projet
- Meilleures pratiques
- Rechercher un texte (Find Text)
- Get Color At Position
- Obtenir la position du curseur (Get Cursor Position)
- Obtenir le champ (Get Field)
- Obtenir le champ en position (Get Field at Position)
- Accéder à la zone d'écran (Get Screen Area)
- Obtenir le texte (Get Text)
- Obtenir le texte en position (Get Text at Position)
- Déplacer le curseur (Move Cursor)
- Move Cursor to Text
- Envoyer la touche Contrôle (Send Control Key)
- Envoyer les touches (Send Keys)
- Envoyer les touches en toute sécurité (Send Keys Secure)
- Définir le champ (Set Field)
- Définir le champ en position (Set Field at Position)
- Session de terminal (Terminal Session)
- Attendre le texte de champ (Wait Field Text)
- Attendre le texte d'écran (Wait Screen Text)
- Attendre le texte en position (Wait Text at Position)
Using mobile automation for mobile banking application
In this tutorial, we explore how to use the Mobile Automation package to automate tasks within a mobile banking application. The tutorial uses the mobile banking simulator provided by Bank of America, and a Google Pixel Emulator from Sauce Labs. We will specifically focus on automating and testing the following actions:
- Logging into the app.
- Implementing a debit card lock.
- Viewing a claim and capturing a screenshot.
The tutorial provides instructions on how to use both activities and APIs for automating these scenarios.
- Add a device from Sauce Labs into MDM.
For more information about adding devices, visit Adding Sauce Labs device.
- Add the Bank of America mobile banking simulator into MDM.
For more information about adding applications, visit Adding applications.
To follow the steps and try out the tutorial yourself, check the following sample project: Mobile Banking Simulator.
In this example we use Object Repository elements as input UI elements to activities. To create UI elements for mobile, establish a connection in Mobile Device Manager (MDM) between the target device and application. Then, directly in the Object Repository panel, create each element and use Indicate Element to indicate the specific UI element on the MDM device.
To easily follow the tutorial, check the following list for the Object Repository structure:
- BankOfAmerica application, MobileScreen menu
- backFromClaimDetails
- backFromClaimStatus
- BackToMenu
- firstClaimSeeDetails
- Lock
- LockCard
- Login
- ManageDebitCredit
- Menu
- Mot de passe (Password)
- swipeMenu
- UserId
- viewmyclaims
- Establish a connection between the device and application:
- Add a Mobile Device Connection activity.
- Use Select Connection Details to connect the previously created mobile device and application.
- Login to the mobile application:
- Add a Tap activity to select the User ID field, so you can then enter a username.
- Add a Set Text activity to input the username inside the User ID field.
- Add a Tap activity to select the Password field, so you can then enter the password.
- Add a Set Text activity to input the corresponding password.
- Use a Tap activity to select the Login button.
Figure 1. The first activities as part of logging into the application
- Lock a debit or credit card:
- Add a Tap activity to navigate to the mobile banking app menu.
- Add a Swipe activity to swipe down in the menu.
The goal is to swipe until you find the Manage Debit/Credit section.
- Add a Tap activity to select the Manage Debit/Credit section.
- Add a Tap activity to select the Lock icon for the first card that is displayed.
- Add a Swipe activity to swipe down in the lock card notice.
This action allows you to confirm the action by selecting Lock Card.
- Add a Tap activity to select Lock Card.
Figure 2. Activities used for locking the debit card
- Take a screenshot of a claim:
- Add a Tap activity to select the back button, and return to the main menu.
- Add a Swipe activity to swipe down in the main menu, until the View My Claims section appears.
- Use a Tap activity to select the View My Claims section.
- Add a Tap activity to select See details for the first claim.
- Use the TakeScreenshot API to take a screenshot of the first claim details.
Figure 3. Activities used to take a screenshot of a claim
- Finally, after taking the screenshot, we go back to the main menu of the app, by using the Tap activity to select the Back UI element twice.
To automate a mobile application using APIs, record the application mobile elements within the project's Object Repository. For this example, we created a BankOfAmerica application with a general MobileScreen, containing all the UI elements that we use to automate. To create UI elements for mobile, establish a connection in Mobile Device Manager (MDM) between the target device and application. Then, directly in the Object Repository panel, create each element and use Indicate Element to indicate the specific UI element on the MDM device.
Check the following list for the structure of the Object Repository:
- BankOfAmerica application, MobileScreen menu
- backFromClaimDetails
- backFromClaimStatus
- BackToMenu
- firstClaimSeeDetails
- Lock
- LockCard
- Login
- ManageDebitCredit
- Menu
- Mot de passe (Password)
- swipeMenu
- UserId
- viewmyclaims
- Establish a connection between the device and application:
- Create a variable of type
Connection
, using the Connect API, with the second overload version.For this overload, enter the device name and application name as Strings, to establish the connection.
[TestCase] public void Execute() { Connection connection = mobile.Connect( "Android mobile device", "BankOfAmericaSimulator"); Log("Connection established");
[TestCase] public void Execute() { Connection connection = mobile.Connect( "Android mobile device", "BankOfAmericaSimulator"); Log("Connection established");
- Create a variable of type
- Optionally, for debugging purposes, you can add a
Log
message after the connection. - Log in to the mobile application:
- Call the Tap API to click the User ID field.
This action allows you to enter the username.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId); - Use the SetText API to enter a username in the User ID
field.
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId, "john.doe@uipath.com", MobileOptions.SetText().WithSendNewline(false));
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.UserId, "john.doe@uipath.com", MobileOptions.SetText().WithSendNewline(false)); - Call the Tap API to
select the Password field so you can enter the
password.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password); - Use the Set Text API to enter a password in the Password field.
The
WithSendNewLine(false)
method ensures that the OK keyboard element is not pressed on the mobile device after sending the text. For more information about the available options of the Set Text API, visit MobileOptions.SetTextOptions Class.connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password, "abcde", MobileOptions.SetText().WithSendNewline(false));
connection.SetText(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Password, "abcde", MobileOptions.SetText().WithSendNewline(false)); - Employ the Tap API to
select
Login.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Login);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Login);
- Call the Tap API to click the User ID field.
- Lock a debit or credit card:
- Navigate to the menu of the mobile banking app, using the Tap
API:
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Menu);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Menu); - Swipe down in the menu, using the Swipe API.
The goal is to swipe until you find the Manage Debit/Credit section.
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, SwipeDirection.Down, 4);
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, SwipeDirection.Down, 4); - Select the Manage Debit/Credit section using the Tap API.
The
WithTimeout()
method enables the API to pause for a predetermined number of seconds. This pause lasts until the target UI element appears, or an error is thrown.connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, TapTypeEnum.Single, MobileOptions.Tap().WithTimeout(120));
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.ManageDebitCredit, TapTypeEnum.Single, MobileOptions.Tap().WithTimeout(120)); - For the first card that
is displayed, select the Lock icon using the Tap
API.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Lock);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.Lock); - Swipe down in the lock
card notice, using the Directional Swipe API, so we can confirm the action by
selecting Lock
Card.
connection.DirectionalSwipe(SwipeDirection.Down);
connection.DirectionalSwipe(SwipeDirection.Down); - Select Lock Card using the Tap
API.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.LockCard);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.LockCard);
- Navigate to the menu of the mobile banking app, using the Tap
API:
- Take a screenshot of a claim.
- Go back to the main menu,
using the Tap API and the back
icon.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.BackToMenu);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.BackToMenu); - Swipe down in the main menu, until the View My Claims section appears.
For this step, we use the Swipe API, this time employing the sixth overload. This overload requires an
IElementDescriptor
for the swipe container, and aSelectorTarget
for the target UI element.We first create aSelectorTarget
for the View My Claims section, and then employ the Swipe API.- Open MDM and create the connection between the mobile device and application.
- Go to the Object Repository and create a new element for the mobile screen.
- Select Indicate element.
- From the selection options menu, copy the web control identified by the selectors.
For this mobile banking app, you have to ensure that WebView selectors as native selectors is selected as a selection option.
- Return to the
code editor, and create a variable of type
SelectorTarget
. Use theMobileTarget.FromSelector
method which requires as argument the web control you previously copied.SelectorTarget viewMyClaimsTarget = MobileTarget.FromSelector("<mbl android:className='android.widget.TextView' text='View My Claims' />");
SelectorTarget viewMyClaimsTarget = MobileTarget.FromSelector("<mbl android:className='android.widget.TextView' text='View My Claims' />"); - Use the Swipe
API, with the
SelectorTarget
variable, to finish the swipe down action.connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, viewMyClaimsTarget, SwipeDirection.Down, 20);
connection.Swipe(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.swipeMenu, viewMyClaimsTarget, SwipeDirection.Down, 20);
- Tap the View My
Claims
section.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.viewmyclaims);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.viewmyclaims); - Tap See details
for the first
claim.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.firstClaimSeeDetails);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.firstClaimSeeDetails); - Use the TakeScreenshot API to take a screenshot of the first claim
details.
connection.TakeScreenshot();
connection.TakeScreenshot();
- Go back to the main menu,
using the Tap API and the back
icon.
- Finally, after taking the screenshot, we go back to the main menu of the app, by
tapping the Back UI element
twice.
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimDetails); connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimStatus);
connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimDetails); connection.Tap(ObjectRepository.Descriptors.BankOfAmerica.MobileScreen.backFromClaimStatus);