Test Suite

latest

false

Test Suite User Guide

Last updated Apr 22, 2024

Executing Tests

You can execute tests manually or automated. Manual test executions are invoked through Test Manager, whereas automated executions are invoked through Test Manager and executed via Orchestrator. Additionally, within one test set execution, each individual test case can be executed either manually or automated. This changes the execution type to Mixed.

To execute your test cases, you must first create a test set, which will be later executed. If you want to execute test cases individually, instead of executing all of them at once, you can create a pending execution from a test set. From any execution, you can choose to execute individual test cases either automated or manually.

Conditions

You can use both Execute Manually and Execute Automated for test sets created in Test Manager as well as the linked test sets from Orchestrator.
Only the test case versions included in the test set are executed. If you want to execute new test cases, you need to assign new versions to your test set. For Orchestrator test sets, you need to make sure that the latest packages are published.

Automated execution

Automated execution refers to test sets that have been linked from Orchestrator or test sets created in Test Manager that have test cases linked from Studio.

Manual execution

Manual execution refers to test sets created in Test Manager, including test cases that have manual steps. Manual execution can be done for test sets created in Test Manager as well as the linked test sets from Orchestrator.

Executing Manual Tests

Execute Test Sets Manually

Both Test Manager test sets and test sets that have been linked from Orchestrator can be executed manually. if manual steps have been defined for the test cases. For more information, see Executing Tests.

Open your project in Test Manager and then navigate to Test Sets.
Select a test set and click the vertical ellipsis.
Select Execute Manually. The Manual execution assistant opens in a new window. Make sure your popup blocker allows the window to be opened.

For more information on test results, see Analyzing Test Results.

Note:

Test cases that do not contain any manual steps will be skipped during execution without producing any result.
Automated test case can have manual steps defined and can also be executed manually.

Alternatively, you can open your test set, click Execute and select Manually from the dropdown.

Manual execution assistant

The manual execution assistant lets you perform a set of manual tests one after each other. You need to set a result on each manual step to advance to the next test case in the test set.

Through the Manual Execution Assistant, you need to treat and mark steps in each case either as passed or failed.

Use Set Failed or Set Passed to mark a step in each case. Add comments, especially in case of a failure, to provide details about the particular result. You can set several consecutive steps as passed by clicking Set Passed Till Here.

Once the last step of a test case has a result assigned, you can click Next. This enables you to go to the next test case from the test set. Once you are finished with the last test case, clicking Done finishes the execution entirely.

Executing Automated Tests

You can execute all test sets using Execute Automated. Orchestrator is invoked for test cases that have been linked to Test Manager. To execute test sets linked from Orchestrator, make sure you create your test sets in Orchestrator.

You can also execute automated test if the test set includes test cases linked from Studio. For more information, see Executing Tests.

Conditions

Important: To use the Execute Automated option, you need at least one automated test case in a test set.

Automated test executions have a default timeout of 12 hours. Any automated test execution that exceeds 12 hours is marked as Cancelled.
Test execution for test sets linked from Orchestrator includes only the test cases that have been published to the Orchestrator Tenant Process Feed in the latest package version. If you want to execute new test cases, you need to publish a new package version in Orchestrator, and then link the test set to Test Manager.
If your Test Manager test set includes test cases from multiple Studio projects, then the project that has the highest number of test cases is considered for execution. Therefore, only the test cases from this project will be executed. In this case, you will receive the following message within Test Manager: "Test cases from multiple packages have been selected. Some of the test cases will not execute while running this test set."
You can execute test sets including test cases from multiple Studio projects, only if you have a test set linked from Orchestrator.
If your test set includes test cases that are duplicate in multiple Studio projects, the sorting is done by the project name in descending order. Only the test cases from the project that is placed first in the list will be executed.

Execute Automated Test Sets

Open your project in Test Manager and then navigate to Test Sets.
Select a test set and click the vertical ellipsis.
Select Execute Automated.
For more information on test results, see Analyzing Test Results.
Note:
- Test cases that cannot be executed are skipped during execution through Orchestrator.
- Test cases that are not executed generate test case logs with no result.

Alternatively, you can open your test set, click Execute and select Automated from the dropdown.

Troubleshooting

Executing mixed tests

You can execute mixed tests by running both manual and automated tests again from a past test result. When you re-execute mixed tests, it always uses the latest version of a test case.

Execute mixed test sets

Open your project in Test Manager and go to Test Results.
Select a test set with both manual and automated test cases.
Select both types of test cases and then select Execute Automated.
Observe the test result's Execution type switch from Manual or Automated to Mixed.

Creating pending executions - Preview

A pending test execution is similar to a regular test execution made during a test set run, the only difference being that no tests are run immediately. A pending execution assigns the None result for each test case from the test set, allowing you to start tests later or to execute them in chunks or individually.

When you can use pending executions

When you want to execute tests at a later time or not all at once, but still reflect the current state of the test set.
When some tests are run manually while others are supposed to run automated.
When you want the executed tests to reflect the current state of a test set including the selected test cases.

A test execution is independent of its originating test set, with all assigned test cases becoming static members of the execution. To ensure a specific set of tests from a test set is used in an execution, even when the test set is updated, create a pending execution.

Steps to create a pending execution

Open a project in Test Manager, and navigate to Test Sets.
Select Tasks and then Create Pending Execution. This creates a replica of the test set that is ready for execution. The pending execution is opened automatically and takes you to Test Results.

Enforcing an execution order

You can set a specific execution sequence for test cases within a test set or a test execution. This sequence determines the order in which the test cases run initially. However, any changes to the order within a test execution do not affect the order in the test set. So, the sequences you set within a test set and a test execution operate independently of each other.

Note: You can enforce a specific execution order only for statically assigned test cases, with dynamically assigned ones following in a random order only after the static ones have been completed.

Steps

Determine where you want to reorder the test cases within your project:
- Within a test set.
- Within a test execution.
Enable the Enforce Execution Order toggle.
- Go to Test Sets, and then to the Configuration tab, and select the Enforce Execution Order toggle.
- Alternatively, if you want to reorder within a test execution, go to Test Results, and then to the Settings tab, and enable the Enforce Execution Order toggle.
After you've enabled the execution order, return to the tab where you see the test cases. Depending on where you've activated the enforcement, follow these steps:
- If you're within Test Sets, return to Test Cases and select Reorder.
- If you're within Test Results, return to Results and select Reorder.
This opens the Reorder test cases view where you can arrange the order of test case executions.
Drag and drop test cases to reorder them in the test set or test execution.
When you finish reordering the test cases, select Save.

When you execute the test set, the test cases will run in the order you previously saved.

Re-executing test executions

If you re-execute a test execution the old execution results are deleted and replaced with the new results. If you don't want to replace a test exection result, then run the test set again. This will generate new test execution results, independent from the previous ones.

The pre-defined user roles that can perform re-executions are: Test Manager, Test Executor and Manual Tester.

Re-executing test set executions

Go to Test Results and choose one of the following:
- Select More Options for the test execution that you want to redo.
- Open the test execution and select Tasks.
Select one of the following:
- Re-execute manually (available only for manual test executions)
- Re-execute automated (available only for automated test executions)
The Re-execute dialog appears.
Select Confirm and execute the manual steps or wait for the automated test set to finish.

Note:
Re-executing test set executions doesn't display the activity coverage anymore.

Re-executing test case executions

You can re-execute one or more test case executions. For data-driven test cases, you can only re-execute one test case execution at a time. You can't re-execute data-driven test case executions in bulk.

Note: Even if you publish a newer version of the test case, the version that you re-execute is the same version that is recorded in the test execution.

Go to Test Results and open a test set execution.
Select the checkboxes for one or more test case executions and then select Re-execute selection.
Select one of the following:
- Re-execute manually (available only for manual test executions)
- Re-execute automated (available only for automated test executions)
The Re-execute dialog appears.
Select Confirm and execute the manual steps or wait for the automated test case to finish.

Troubleshooting Automated Executions

Automated execution could not start since the package was not found on Orchestrator. Only test cases from packages which have been published to Orchestrator can be executed.

Solution

You need to publish the Studio package to Orchestrator. This issue occurs when a test case is linked from Studio to Test Manager but the package is not yet published to Orchestrator.

Test cases could not be executed since no executable versions could be found. Only test cases published with the latest package version to Orchestrator can be executed.

Solution

This issue occurs when test cases in a test set are not published in the latest Studio package to Orchestrator. You might get the same error when there is no process/release found on Orchestrator for the studio package.