v24.10

v24.10.1

发行日期：2024 年 7 月 1 日

Autopilot Recorder for UI Automation

We are excited to introduce our latest feature, the Autopilot Recorder for UI Automation. This AI-powered tool is available in Studio Web and allows you to build UI Automation sequences and populate them into your canvas. You can generate activities via a natural language prompt or manually add them in the Recorder.

The Recorder allows you to navigate through the application you want to automate using the Test and Continue 'prompt' buttons. The Test button executes all current activities you have generated or built, transitioning you to the next application state. If your prompt is not fully covered, the Continue 'prompt' generates the necessary activities to complete it. Working together, these two buttons enable a seamless progression through your process of scaffolding UI Automation sequences.

For more details about the Recorder, refer to the Autopilot Recorder for UI Automation page.

For an extensive overview of the Autopilot initiative, check out the Autopilot guide.

v24.10.0

发行日期：2024 年 7 月 1 日

常规改进

• 添加了对BasicTreeTable和JxTreeTable Java 控件的自定义支持。
• ClosePopup, SetRuntimeBrowser, and SetValue activities can now be used from coded workflows.
• To enhance security, the output .xaml file does not include the FriendlyName property anymore.
• Added support in the Extract Table Data activity for DevExpress grid.
• The UI-USG-011 Workflow Analyzer rule now ignores selectors with idx=* because this is a valid case used in non-greedy search.

用户界面自动化新式活动

When running in Attended mode or debugging in Studio, if the execution stops due to a UI element not being found, a dialog box is displayed on the screen. Previously, only the error message was shown. Currently, several options in the dialog box allow you to decide what to do next. This functionality can be enabled via the Ask user on runtime error project setting (UI Automation Modern > Generic). In Debug mode, you can Save the changes when the execution ends, so that the next workflow run starts directly with the correct selector. In addition to the Retry, Ignore, and Continue options, you also have the option to Indicate the element. This particular option is available only for the following activities:

• 选中/取消选中
• 检查元素
• 单击
• 获取属性
• 获取文本
• 高亮显示
• 悬停
• 插入 Js 脚本
• 键盘快捷方式
• 选择项目
• 设置文本
• 截取屏幕截图
• 键入
• 使用应用程序/浏览器

The new Set Runtime Browser activity allows you to easily configure the browser type used throughout your automation executions. The selection remains effective until the execution ends unless overwritten by another Set Runtime Browser activity. This enhancement simplifies the process of running automations across different browsers, minimizing the need to manually configure selectors and to duplicate testing code. The activity is available in both Windows and cross-platform projects.

The new Close Popup activity is designed to address automation disruptions from unexpected pop-ups. This AI-based activity efficiently closes any detected pop-ups, whether they are native, alerts, found on a browser page, or within a desktop application by semantically matching the labels configured for possible close buttons. The activity is available in both Windows and cross-platform projects.

The Inject Js Script activity is now supported in cross-platform workflows. Additionally, this activity can now leverage the power of UI Automation Modern to identify targets through Unified Target and use stored elements from Object Repository.

The Check App State activity now allows you to select the Indicate in any app option while inside a Use Application/Browser activity, avoiding the need for a new application card when checking the application states for windows or elements outside the scope of the current application card.

Exposed the Arguments property in the Use Application/Browser activity for cross-platform projects. The property was already available for Windows projects.

The following Workflow Analyzer rules, previously exclusive to Classic activities, now apply to Modern activities as well:

• UI-ANA-016
• UI-ANA-017
• UI-DBP-013
• UI-PRR-004
• UI-REL-001
• UI-SEC-004
• UI-USG-011

To help you address the matching issue in the event of a target search failure with fuzzy selector enabled as a targeting method, a warning message highlighting the closest matches is displayed on the screen. The closest matches are not listed if you altered the default selector by adding or removing selector nodes or if the search failed because of multiple matches.

Added a Do not show again checkbox in the Target application could not be identified dialog box. When selected, if the application is not in the correct state and you want to edit the target, the dialog box is no longer displayed and you are taken directly to the selection screen.

Optimized the overlapping behavior for targets and anchors by narrowing the rectangle border and positioning the labels where they can be clearly seen.

Duplicate targets are now displayed in the selection screen on demand. To check/hide duplicate targets, select the Show/Hide duplicates toggle button.

Enhanced the App/Web Recorder functionality by customizing available activities based on element types. Now, different activities appear when indicating elements, making it faster to find and select relevant actions. The following activities are available:

• Default to all elements:
  • 单击
  • 截取屏幕截图
  • 高亮显示
• For labels:
  • Get text
• For editable text boxes:
  • Get text
  • Type into
• For drop-downs and list boxes:
  • Select item
  • Get text
• For buttons:
  • 单击
  • Get text

You can now check the current value of a variable in the selection screen without having to access the variable context menu. To do so, simply hover over the variable from the selector tag and the value is displayed as a tooltip.

基于触发器的 Attended 自动化

Added in the Application Event Trigger activity two new types of events that can be monitored for WND, CTRL, and UIA UI elements: Click and the Key pressed. Until now, the Click and the Key pressed events could be monitored separately by using the Click Event Trigger or the Keypress Event Trigger activity, respectively.

Enhanced support to include monitoring of Edge IE Mode native events, primarily assisting legacy applications that can only run on Internet Explorer. All existing web events for WEBCTRL elements are now also supported on browser pages utilizing Edge IE Mode via the Application Event Trigger activity.

Building upon the functionality of automating standalone applications with native embedded browsers such as Slack, Discord, or Electron, we have incorporated support for native monitoring inside managed embedded browsers, such as CefSharp, in this release.

The support for monitoring the Text selection changed event was added for application built on the Microsoft UI Automation framework, such as Word, Outlook, Excel, or OneNote.

Added HTML native monitoring support for web pages with active CSP loaded using WebView2 or Qt WebView widgets embedded in native applications. The Enable WebView2 Native Automation project setting needs to be set to True.

Added several HTML window events for a web page identified via <html> + <webctrl> selectors:

• 已开始选项卡导航
• 已完成选项卡导航
• 窗口焦点已更改
• 窗口边界已更改
• 窗口已创建
• 窗口已删除

语义活动

通过我们在 Windows 和跨平台项目中提供的最新语义活动，体验表单自动化的简单性和有效性。

The Extract Form Data activity is designed with form data in mind. It creates custom type objects that are ideal tokens for passing structured data along your automation pipeline. These custom type objects can also be used in the Fill Form or Set Value activities or for populating data entities.

The Fill Form activity can use as input any structured data object: DocumentData, Dictionary, DataEntity, DataRow, DataTable, JSON file, even custom types, like those produced as output by the Extract Form Data activity. The data schema is automatically extracted at runtime and AI semantically matched with the destination form fields, which is then inputted into the corresponding form. You just need to indicate a data source and the destination form is automatically picked up from the Use Application/Browser activity.

The Set Value activity is a simplified version of the Fill Form activity, focusing on just one UI element. This activity simplifies the process of data entry. Regardless of the form element, whether it is a text box, a drop-down, a checkbox, a radio button, and so forth, this activity adapts to the element type, enhancing your form automation process. It can even deal with date pickers and groups of radio buttons.

Computer Vision

Computer Vision has successfully transitioned into cross-platform projects and it is now fully available in Unified Target. This represents a significant step forward as it allows you to automate scenarios without selectors, not just in Windows-based projects, but also in Studio Desktop cross-platform projects and in Studio Web.

This release introduces our new Vision Transformer AI model. Due to the new Vision Transformer architecture, UI elements detection in Computer Vision-based automations is faster and more precise across the board. This leads to less errors and speeds up your automation workflows. It particularly brings noticeable improvements in table detection, supporting various table structures and sizes. Due to higher GPU requirements, this model is available only on the UiPath Cloud Server.

Added full Computer Vision support to the Select Item activity, enabling not only the scenario of pure Computer Vision, but also the hybrid scenario of selector and Computer Vision. When selectors are not available (a pure Computer Vision scenario), the activity now replicates the functionality of the CV Dropdown Select activity. When selectors are available (a hybrid scenario), but fail during runtime, Computer Vision triggers a self-healing process, performing a new selector search based on Computer Vision information. This is then used for execution. Furthermore, a warning message with detailed information about the problematic selector and the self-healing outcome is displayed in the logs, empowering you to fix automation issues using newly discovered selectors.

The Get Attribute activity now also supports radio button targets, not also checkboxes. This is done by using a new attribute, checked, which returns a Boolean value (True or False) according to the Computer Vision AI model detection.

You can define a custom point where the scroll event is sent to the target element. Before, the scroll event was sent in the middle of the target element. Now, this can be done by using the run-time auto-scrolling feature in the Computer Vision activities via the Scroll Offset property in the CV Screen Scope activity and, when extracting scrollable tables or scrollable text, by using the Content Scroll Offset property in the CV Extract Table and CV Get Text activities.

When the Fuzzy selector targeting method fails to find an auto-anchor, Computer Vision will now step in and provide an auto-anchor generated by the AI model.

To switch from the default UiPath Screen OCR engine to the UiPath Extended Languages OCR engine in the Use Browser/Application and CV Screen Scope activities, you must provide the Computer Vision API key. This can be done either in the ApiKey property, at activity level, or under Project Settings > OCR > UiPath Extended Languages OCR > ApiKey.

出于计算机视觉运行时分析的可追踪性和调试速度的目的，现在在 HTTP 请求的标头中发送设计时屏幕 ID。

SAP

Studio 中提供了新的项目设置： “项目设置” >“用户界面自动化传统” >“ SAP ” >“启用 SAP 监控”。 此设置允许链接已执行的事务和测试用例。 当“启用 SAP 监控”设置为True时，系统将在运行时收集有关已执行事务及其依赖项的信息，并将其发送到 Test Manager。 此设置当前仅适用于 Studio 中的测试自动化项目。 未来的版本将添加对其他项目类型的支持。

我们添加了对新的 SAP WebGUI 元素的支持：SAP 状态栏。 现在，您可以使用“ SAP 读取状态栏”活动从 SAP 状态栏中提取消息类型和文本。

浏览器扩展程序

为了响应 Google 从 2024 年 6 月开始弃用 Chrome Manifest V2 (MV2) 的时间表，适用于 Chrome 和 Edge 的现有 Studio Web 扩展程序已更新为与 Manifest V3 完全兼容，同时为这两种浏览器保留相同的扩展程序 ID。 新的 MV3 扩展程序不需要执行任何操作，当您重新打开浏览器时，这些扩展程序将自动更新。

如果您将 Chrome 和 Edge 扩展程序升级到 24.10，则在通过组策略安装扩展程序时，不再需要手动将“开发者工具可用性”策略值设置为 1。

Safari 用户界面自动化支持现已在预览版中提供。 全新的 UiPath Safari 扩展程序允许您使用 macOS 上的 Safari 在 Studio Web 中设计用户界面自动化，并通过 macOS 上的 UiPath Assistant 和 Robot 在 Safari 中执行浏览器自动化。

对象存储库

We've made substantial improvements to our Object Repository, adding functionality for previous unsupported scenarios. The following paragraphs provide a comprehensive rundown of each improvement.

Support for Object Repository, previously exclusive to Windows, has now been expanded to include cross-platform in Studio Desktop. Being able to also create and reuse UI objects in cross-platform type projects allows for a faster automation development process, lower maintenance costs, and easier adjustments for automations impacted by changes in the selector structure of application or UI elements.

Leveraging the power of Clipboard AI's advanced UI extraction engine that merges Computer Vision and DOM extractor, the Capture All functionality is now able to capture all elements in a web page, including scrollable content and hyperlinks. Having relied prior to this solely on the Computer Vision model for element detection, it could extract only the visible elements. Now, with the addition of the DOM extractor, it can extract all elements, not just the visible ones, allowing them to be added to the Object Repository library.

You can now highlight an extracted element while maintaining the current selection in the Capture Elements window using the Highlight icon.

Simplified the user experience for creating, integrating, and editing screens, nodes, and applications within Object Repository, ensuring a more efficient process from start to finish. Before, the process required you to manually navigate through numerous confirmation screens. Now, Object Repository automatically assigns your new object to the appropriate application or screen. If there are any missing parent objects, Object Repository creates them automatically.

In case you already created an automation without Object Repository and you want to add the activities and targets to an Object Repository library, you can use the new Synchronize Object Repository wizard. This synchronizes all the UI interacting activities and creates an application with its the name, a screen with its name, and the corresponding UI elements into the local repository. In the past, you would have to manually add each application, screen and element to the Object Repository. Now, the entire project is added to the Object Repository local library at once. This is available both for Modern and Classic activities. For Classic activities, only the targets that are using full selectors are added, while those with partial selectors are not. Moreover, you can now include an existing library and the matching objects that you indicate are going to be synced automatically.

When an activity is linked to Object Repository, edits made to a local object in the target properties panel now automatically propagate to the corresponding descriptor in Object Repository, removing the need for manual synchronization.

Target properties of activities linked to read-only objects are automatically disabled. Before, these target properties would stay active in the properties panel, potentially leading to confusion. Now, if you want to edit the target properties, you have the option to unlink the object and make local edits. However, it should be noted that these changes do not propagate to the actual descriptor in Object Repository.

We've standardized the burger menu of UI Automation activities, aligning the options of activities which have targets linked to a local Object Repository with the options of regular activities. Previously, certain options, such as Indicate target on screen or Edit target, were unavailable for Object Repository-linked activities. We've also standardized the indicate functionality for Use Application/Browser activities when Object Repository is either enforced or non-enforced.

错误修复

• UI Explorer failed to validate selectors for SAP Interaction Center when attempting to bypass the use of iframe.
• Unexpected behavior of terminal Java application when run with a 32-bit executable; now works smoothly with both 32-bit and 64-bit executables.
• 导航至 Kendo Angular 表格的第二页时，“提取表格数据”活动未能成功抓取数据。
• 修复了从包版本 21.10.6 开始发生的 Java 应用程序弹出窗口问题，该问题影响了设计时体验。
• 当项目名称包含日语字符时，中文、日语和韩语 OCR 在“使用应用程序/浏览器”活动中不能作为自定义 OCR。
• 指定目标后，从计算机视觉屏幕作用域中删除 OCR 引擎活动时，引发 Value cannot be null 异常。
• 从计算机视觉屏幕作用域中删除 OCR 引擎活动时引发 There is no row at position 0 异常，并且悬停在表格单元格上方的操作持续存在。
• Elements on web pages that were recognized by the selector widget were not preserved upon saving and closing the selector.
• Unable to interact with embedded web widgets in applications that host web controls in multiple top-level windows.
• Exceptions thrown by cancelling a Click activity in a parallel sequence weren’t caught by a Try Catch.
• Extracting large table data from static HTML pages was causing errors and slowing down all supported browsers. This fix doesn’t cover Internet Explorer.
• Previously, the Get From Clipboard activity might not fetch the most recent value from the clipboard.
• The Type Into activity would ignore the text inserted before special characters. This behavior occurred when the Input mode property was set to Simulate.
• The Select Item activity and the SAP-specific Select Menu Item activity were failing when the selected item contained text with wildcards.
• Indicating lines in a notepad application would fail.
• When using Foxit reader, a few issues were encountered. The Extract Table Data activity could not extract Chinese characters from scanned PDF files. Additionally, UI Explorer could not trigger document accessibility. Furthermore, even though running an automation in attended mode functioned correctly, searching for the application during unattended mode failed.
• When indicating the document name from the ribbon of an open Word document stored on shared drives, such as SharePoint, you would encounter an invalid selector error.
• HTML elements in nested iframes on Salesforce did not auto-scroll into view when a mouse input is sent via Hardware Events.
• In some cases, UiPath Remote Runtime takes longer to return Get Text results from a remote machine. As a workaround, set Windows attach mode to Single window instead of Application instance in the Use Application/Browser activity. Note that the Single window option cannot interact with new application windows or pop-ups. Use a new Application/Browser activity for these interactions.
• When upgraded from Windows - Legacy to Windows projects, the Extract Table Data activity could not extract column values due to incorrect selector generation.
• Upon downloading the latest WebDriver version, the Open Browser activity stored it in the WebDriverExe folder, but still referenced the previous version instead of applying the current one. An incompatibility error message would appear when trying to open browsers.
• When adding the On Element Appear, On Element Vanish, On Image Appear, and On Image Vanish classic activities to the designer panel, you would get validation warnings.