- Getting Started with Test Suite
- Studio
- Orchestrator
- Testing robots
- Test Manager
Test Suite User Guide
Analyze test results
To view test executions, open Execution in Test Manager. The execution of each test set is listed as a test execution entry. To understand how the test executions work behind the scenes, see Execution.
By analyzing your test execution, you can take the following actions:
- Find test results that have been executed manually or automatically through Orchestrator.
- Check the progress on running test executions.
- Open the test set that has been executed.
- Examine logs and attachments.
- Create defects in your defect management system directly from Test Manager, if you have an Application Lifecycle Management tool integration. For more information, see ALM Tool Integration.
By default, the test executions are sorted based on the date on which the execution was finished. Currently running and pending executions are placed at the top.
You can identify the status through the color codes assigned to each test execution, as follows.:
- Green: Test cases that passed.
- Red: Test cases that failed.
- Grey: Test cases without a definitive results, such as test cases that have not been executed yet but are part of a test set that is currently being executed.
The results from automated test executions are imported from Orchestrator automatically. To have your automated tests imported to Test Manager, you need to meet the following conditions:
- The automated test needs to be
part of a test set on your Test Manager project. For more information, see Automated Tests.
Note: If a Test Execution from Orchestrator holds results for test cases from several projects in Test Manager, the test execution is split in Test Manager. The results appear in the projects where the test cases are.
Overriding the results of a test case can help in scenarios where the current test result does not accurately reflect the actual behavior of the application and re-execution is not an effective solution. You can override the test results of a test case and then you can clear the operation, if needed.
- Navigate to Execution and open a test result.
- From the Results tab open a test case log.
- In the Assertions tab, select Tasks > Override result.
- In the Override test
result window configure the following fields:
- Change result to - select whether you want to override the result with the opposite status (Passed/Failed) or set the result to None.
- Comment - Type the reason why you override the result.
- Select Confirm.
The test results that you overrode have the following icon next to the test result status:
. Select the icon and you can view the override details.
Figure 1. The "Test result overridden successfully" message
- Optionally, if you want to clear
the override operation, select the
- In the Override
details window, select Clear override.
Figure 2. The Override details menu
- In the Override
details window, select Clear override.
- Optionally, if you want to edit
the override operation, select the
- Perform changes, and click Confirm.
To quickly find your test results, use the search function and the filters. Navigate within the page using and configuring the paginator. Alternatively, you can use the breadcrumb to navigate between the pages.
- Filters - You can use the
filters to narrow your search. For example, you can search for test results by
the following criteria:
- Labels
- Execution Type
- Execution Finished
- Status
The filters are automatically saved and kept active until you clear them.
- Search - Use the search bar to find test results by their key, execution type or status (requires full search term match). You can use the search bar at the top of the page to look for test results.
To view detailed test results, go to Execution.
- Test Set: Test results are grouped by Test Execution (Test Set).
- Test Case: For
cross-execution analysis when filtering the results based on criteria
excluding test execution attributes.
You can access the execution log of a test case from the Test Case view by selecting the relevant result chip. If the postcondition of a test case wasn't met during a manual execution, the test case log displays an exception with an Info icon. The exception message reads: Exception during execution. View logs for details. Selecting this message takes you to the Execution details tab of the corresponding test case log.
You can use the Reporting Date filter to set a unified date across all test results from the test execution. This is useful when you run tests overnight, where some test cases are executed before and after midnight, preventing results from splitting onto separate dates.
In the Results tab you can examine all the executed test cases within the test set, and take action for each, as necessary. For more information, see the Execution log section.
You can also open the executed test set by right-clicking an entry in the Test Sets page and selecting Open Test Set, or directly within the test execution.
As part of test reporting, execution logs hold information such as execution details (e.g., data variation and screenshots), failed and passed assertions, and a detailed log of execution events.
To open a test case log, go to Executions, open a test execution and then select a test case Key, or result.
In the following table, you can view the type of information that is collected during test case execution.
|
Test execution reporting |
Description |
|---|---|
|
Assertions | View failed or passed assertions (e.g.,Verify
expression activity), and associated screenshots
if any were taken during test case execution.
To make sure that
Orchestrator takes screenshots during executions, visit the
following resources:
|
|
Logs | View INFO level logs as part of the
RobotLogs, with information about processes,
execution robot, and event logs, including failures.
Select the
|
|
Execution details | View argument details such as input and output values, as well as execution and robot details (e.g., project, machine, robot). This section also indicates whether the postcondition was met by the test case or not. |
|
Affected requirements | View the requirements that are assigned to the test case that has been executed. You can use this tab to go directly to the affected requirement. |
| Attachments | You can view the following:
|
icon to go to the
Logs tab of the selected test case, for a detailed
description of the failure.
- To download logs for all your test executions: From Execution, select Export to excel.
- To download test case logs for a specific test execution: From Execution, select a test execution, and then select Export to excel.
Visit Export data to check more details about exporting logs to excel files.
The Execution details tab of a test case log offers information about the preconditions ad postconditions of a test case.
If the postcondition was not met during the test execution, then the tab displays the Condition not met warning, that you can investigate.
Prerequisites: Enable activity coverage for the desired test sets either in Orchestrator or Test Manager.
- Open the test set, select More
Options
and then Execute
Automated.
- Go to Execution and open the test set you executed.
- Go to the Activity
Coverage tab and investigate the information on the activities that were
covered during the test execution.
Figure 5. Overview of an example activity coverage
You can create defect reports including the execution log to your external defect management system, if you already have it integrated with Test Manager. For more information, see ALM Tool Integration.
To create a defect out of an execution log, you need to open a test case log, click Tasks, and then select Create Defect. After the defect has been created, a link is available in the execution log, so you can access the integrated external tool.
