Autopilot Recorder for UI Automation

Autopilot Recorder for UI Automation is an AI-powered tool which aims to significantly accelerate and simplify the process of scaffolding basic UI Automation sequences.

You can create UI Automation sequences using the Recorder both by generating activities via a natural language prompt and by manually adding them.

重要:

There is no support for adding loops of conditional logic. The Recorder can only handle a pre-defined list of basic UI Automation activities.

Extract Data Table is not supported yet.

Once you have created the desired sequence, simply select Save. The activities needed to perform the sequence are then populated in your Studio Web workflow canvas.

Accessing the Recorder

To access the Autopilot Recorder for UI Automation, follow these steps:

1. Open an existing Studio Web project or create a new one.
2. In the workflow, add a Use Browser activity.
3. Under Work in scope, select a tab to automate from the drop-down menu.
4. Select the Generate the UI automation activities using Autopilot icon from the top right corner of the informative screenshot box or the Generate button. The Generate button is visible only when there are no activities inside the Use Browser activity.
   
   
   
   
5. The Recorder is now open.
   
   
   
   

Note: If you create a project from scratch with Autopilot and the workflow requires a UI Automation sequence, the part of the text prompt pertaining to the UI Automation sequence is pre-loaded in the Recorder once you select Generate in the Use Browser activity. Therefore, when you access the Recorder in this context, it goes directly into generating the activities.

Generating activities via text prompt

To generate activities via text prompt in the Recorder, follow these steps:

1. Select the plus button.
   
   
   
   

2. Proceed with either step a or b. Regardless of the choice, the outcome is the same.

   1. 1. Select the Generate with Autopilot option from the list. This creates a placeholder activity card.
      2. Write a text prompt in the Describe your actions on this page text field, then press the Enter key or select the arrow symbol. A prompt is most effective if it precisely describes the task at hand. This could include instructions such as "create a new supplier" or "create a time off request between 25 and 30 December", instead of a generic instruction like "click the 'new' button".
         
         
         
         
   2. Write a text prompt in the search box, then press the Enter key or select the Generate with Autopilot option.
      
      
      
      
3. The activities are now generated from the prompt. The Recorder uses only the following UI Automation activities to generate sequences:
   • クリック
   • 文字を入力
   • テキストを取得
   • 項目を選択
   • チェック/チェック解除
4. The generated activities are populated in the Recorder. To transition to the next application state, select the Test button. The Test button is a one-click default-configured execution attempt with the sole purpose of helping you transition to the next application state so that you can continue scaffolding your automation.
   
   
   
   
5. You have transitioned to the next application state. Executed activities are marked with a green checkmark. If, for some reason, the Test default-configured best-effort execution attempt fails, you can still interact with the application freely to manually execute the necessary steps. If the prompt is not covered in its entirety, select the Continue 'prompt' button.
   
   
   
   
6. A new activity has been added in the Recorder. To transition to the next application state, select the Test button.
   
   
   
   
7. You have transitioned to the next application state. You can use the Continue 'prompt' and Test alternatively until you reach the desired application state and the entire prompt is covered.
   
   
   
   

Manually adding an activity

To manually add an activity in the Recorder, follow these steps:

1. Select the plus button.
2. Choose one of the following UI Automation activities from the list:
   • クリック
   • 文字を入力
   • テキストを取得
   • 項目を選択
   • チェック/チェック解除
   • ホバー
   • キーボード ショートカット
   • スクリーンショットを作成
   • 属性を取得
   • マウス スクロール
   • 要素を確認
3. Indicate a target by selecting a UI element on the screen.
4. Indicate an anchor by selecting a nearby UI element that helps to uniquely identify the target.
5. [確認] を選択します。
6. The activity is now added in the Recorder.

Actions you can take in the user interface

Explore the options available in the user interface as detailed in the following table.

オプション	説明
Generate button	Opens the Recorder. You can also select the Generate the UI automation activities using Autopilot icon from the top right corner of the informative screenshot box.
Plus button	Opens the search bar and the drop-down activity list.
Activity card	You can perform the following: Rearrange the activities in the Recorder by drag and dropping them. Access the selection screen by double clicking the activity card. Check the target and anchors of different activities without accessing the selection screen by simply selecting the activity card. This triggers a five-second highlight preview of the target and anchors.
Activity card hamburger menu	This contains the following options: Properties - Opens the selection screen, where you can indicate the target and the anchor. Rename - Enables you to rename the activity. You can use up to 100 characters. Delete - Deletes the activity card from the Recorder.
Test button	Executes all generated activities. This is a default configuration execution meant to help you to transition from one state of the application to another. Note: The Test button is not available in Safari.
Continue 'prompt' button	Generates new activities in the Recorder to cover the entire prompt.
Delete キー	Deletes the selected activity card. Advanced user keyboard shortcut; no confirmation prompt is displayed.
Generate with Autopilot trash bin button	Deletes the Generate with Autopilot placeholder activity card.
Esc キー	Returns to the Studio Web canvas. If you have made changes in the Recorder, a pop-up with the Save your changes made in the Selection Helper? message is displayed. You can select: Cancel - Takes you back to the Recorder, where your changes are preserved. Don't Save - Takes you back to the Studio Web canvas without saving any changes. Save - Saves your changes and your UI Automation sequence is populated in the Studio Web canvas. If the Properties panel is open, pressing Esc takes you back to the Activities view of the Recorder.
Cancel button	Returns to the Studio Web canvas without saving any changes.
Save button	Saves your workflow sequence and populates the activities in the Studio Web canvas.
Help (F1) icon	Opens the documentation page.