Executing tests

You can execute tests manually or automated. Manual test executions are invoked through Test Manager, whereas automated executions are invoked through Test Manager and executed via Orchestrator. Additionally, within one test set execution, each individual test case can be executed either manually or automated. This changes the execution type to Mixed.

To execute your test cases, you must first create a test set, which will be later executed. If you want to execute test cases individually, instead of executing all of them at once, you can create a pending execution from a test set. From any execution, you can choose to execute individual test cases either automated or manually.

Conditions

• You can use both Execute Manually and Execute Automated for test sets created in Test Manager as well as the linked test sets from Orchestrator.
• Only the test case versions included in the test set are executed. If you want to execute new test cases, you need to assign new versions to your test set. For Orchestrator test sets, you need to make sure that the latest packages are published.
• If Execution order is enabled, you cannot run individual test cases from a test set.

Automated execution

Automated execution refers to test sets that have been linked from Orchestrator or test sets created in Test Manager that have test cases linked from Studio.

Manual execution

Manual execution refers to test sets created in Test Manager, including test cases that have manual steps. Manual execution can be done for test sets created in Test Manager as well as the linked test sets from Orchestrator.

Executing manual tests

Assigning manual executions to users

You can distribute your work within a testing project by assigning manual test case executions to users, and setting due dates for them. You can assign manual executions to any user, keeping in mind that they should have permissions to execute tests.

Prerequisites

To assign or unassign manual test case logs, you must have the Change test case executions assignments task permission in one of your roles. By default, this task permission is integrated in the Test Manager pre-defined role.

Assigning manual executions

To assign one or multiple manual executions to a user, perform the following steps:

1. Go to the Execution section of your project.
2. Select the name of a test execution and open it.
   Note: Any test execution can be selected, regardless of its type. This is because the objective is to facilitate manual execution by the user, irrespective of the method used in its prior execution.
3. Assign the manual test case executions in one of the following ways:
   • To assign a single manual test case execution: select More Options for the desired test case, and then select Assignment.
   • To assign multiple manual test case executions: select the check boxes next to the desired manual test case keys, and then select Assign.

   The Assignment dialog box opens.

4. In the Assignee field, enter the username or email address of the user to which you want to assign the test cases.

   Ensure that the user to which you assign the test case has permissions to execute tests.

5. In the Set the due date field, enter the date by which you want the user to run the test case.

   The format of the date is: Month/Day/Year.

   You can also use the date picker for this step.

6. Select Assign to complete your action.

   If subscribed to these events, the user will receive a notification that test cases were assigned to them.

Tip: You can also change the assignment and due date from the header of the test case execution log.

Figure 1. Overview of the Assignment dialog box

Unassigning manual executions

To unassign one or multiple manual executions from a user, perform the following steps:

1. Go to the Execution section of your project.
2. Select the name of the target test execution and open it.
3. Unassign the manual test case executions in one of the following ways:
   • To unassign a single manual test case execution: select the check box next to the desired test case, and then select Clear assignment.
   • To unassign multiple manual test case executions: select the check boxes next to the desired manual test case keys, and then select Clear assignment.

   If subscribed to these events, the user will receive a notification that test cases were unassigned from them.

Scheduling a due date for manual executions

You can set a due date for a manual test case execution. If that date passes without running the test case log, the due date will be highlighted.

To set a due date for a manual test case execution:

1. Go to the Execution section of your project.
2. Select a manual test execution and open it.
3. Open a test case log.
4. In the Due Date field, from the test case log header, enter the due date. The format should be: Month/Day/Year.

   You can also select the date using the date picker.

   Note: You can also set the due date when assigning the manual test case execution to a user.

Execute Test Sets Manually

Both Test Manager test sets and test sets that have been linked from Orchestrator can be executed manually. if manual steps have been defined for the test cases. For more information, see Executing Tests.

1. Open your project in Test Manager and then navigate to Test Sets.
2. Select More Options for the test set that you want to execute.
3. Select Execute Manually.
   The Manual execution assistant opens in a new window. If you are using a pop-up blocker, configure it to allow the window to open.

For more information on test results, see Analyzing Test Results.

Note: Automated test cases can have manual steps defined and can also be executed manually.

Alternatively, you can open your test set, select Execute and then select Manually from the dropdown list.

Manual execution assistant

The manual execution assistant allows you to perform a set of manual tests one after another. You need to set a result on each manual step to advance to the next test case in the test set. Through the Manual Execution Assistant, you need to treat and mark steps in each case either as passed or failed.

Figure 2. Executing manual tests using the Manual Execution Assistant

1. To mark the progress of each step in the test case, select Set Failed or Set Passed.
2. Include comments, especially when marking a step as failed, to supply specific information about the outcome.
3. After marking a step, you can set several following steps as passed. Do this by selecting Set Passed Until Here.
4. Once you conclude a test case by assigning a result to the last step, select Next.

   This action allows you to advance to the next test case within your test set.

5. Repeat steps one to four until you reach the final test case in the test set.
6. If a postcondition is specified for a manual test, verify it after assigning a result to the last step of a test case.
   1. If the postcondition is satisfied, select Condition Met.
   2. If the postcondition is not met, leave Condition Met unselected.
7. After finishing the last test case, select Done to complete the entire execution process.

Executing test cases simultaneously

You can execute manual test cases simultaneously with other users. This allows for the re-execution of test cases even while a manual test set run is in progress. The test case log status is constantly updated according to the most recent user activity encompassing both the start and end of a test. Status updates occur immediately when a user begins to re-execute a previously completed test case log, without preserving the previous result throughout the re-execution process. While a record of past executions is maintained, any new result overwrites the old one, meaning the test case log reflects the outcome of the most recent execution.

Identifying the last test case executor

Within the Results tab of a manual execution, the ExecutedBy column displays the username of the most recent user who executed the test case log. This information refreshes each time you or other users take any of the following actions on a test case:

• Execute a manual test set for the first time;
• Run a pending execution of a manual test execution;
• Re-execute a test case log from a manual execution.

Tracking test case logs

If you re-execute a test case log during a particular test execution, Test Manager will not preserve the test case log's history and will overwrite it. However, you can review the execution history for each test case, across different test executions and associated user details, in the Latest Results tab.

Executing automated tests

You can execute all test sets using Execute Automated. Orchestrator is invoked for test cases that have been linked to Test Manager. To execute test sets linked from Orchestrator, make sure you create your test sets in Orchestrator.

You can also execute automated test if the test set includes test cases linked from Studio. For more information, see Executing Tests.

Conditions

Important: To use the Execute Automated option, you need at least one automated test case in a test set.

• Automated test executions have a default timeout of four days. Any automated test execution that exceeds four days is marked as Cancelled.
• Test execution for test sets linked from Orchestrator includes only the test cases that have been published to the Orchestrator Tenant Process Feed in the latest package version. If you want to execute new test cases, you need to publish a new package version in Orchestrator, and then link the test set to Test Manager.
• If your Test Manager test set includes test cases from multiple Studio projects, then the project that has the highest number of test cases is considered for execution. Therefore, only the test cases from this project will be executed. In this case, you will receive the following message within Test Manager: "Test cases from multiple packages have been selected. Some of the test cases will not execute while running this test set."

  You can execute test sets including test cases from multiple Studio projects, only if you have a test set linked from Orchestrator.

• If your test set includes test cases that are duplicate in multiple Studio projects, the sorting is done by the project name in descending order. Only the test cases from the project that is placed first in the list will be executed.

Execute Automated Test Sets

1. Open your project in Test Manager and then navigate to Test Sets.
2. Select a test set and click the vertical ellipsis.
3. Select Execute Automated.

   For more information on test results, see Analyzing Test Results.

   Note:

   • Test cases that cannot be executed are skipped during execution through Orchestrator.
   • Test cases that are not executed generate test case logs with no result.

Alternatively, you can open your test set, click Execute and select Automated from the dropdown.

Executing mixed tests

You can execute mixed tests by running both manual and automated tests again from a past test result. When you re-execute mixed tests, it always uses the latest version of a test case.

Execute mixed test sets

1. Open your project in Test Manager and go to Execution.
2. Select a test set with both manual and automated test cases.
3. Select both types of test cases and then select Execute Automated.

   Observe the test result's Execution type switch from Manual or Automated to Mixed.

Creating pending executions

A pending test execution is similar to a regular test execution made during a test set run, the only difference being that no tests are run immediately. A pending execution assigns the None result for each test case from the test set, allowing you to start tests later or to execute them in chunks or individually.

When you can use pending executions

• When you want to execute tests at a later time or not all at once, but still reflect the current state of the test set.
• When some tests are run manually while others are supposed to run automated.
• When you want the executed tests to reflect the current state of a test set including the selected test cases.

A test execution is independent of its originating test set, with all assigned test cases becoming static members of the execution. To ensure a specific set of tests from a test set is used in an execution, even when the test set is updated, create a pending execution.

Steps to create a pending execution

1. Open a project in Test Manager, and navigate to Test Sets.
2. Select Tasks and then Create Pending Execution. This creates a replica of the test set that is ready for execution. The pending execution is opened automatically and takes you to Execution.

Enforcing an execution order

You can set a specific execution sequence for test cases within a test set or a test execution. This sequence determines the order in which the test cases run initially. However, any changes to the order within a test execution do not affect the order in the test set. So, the sequences you set within a test set and a test execution operate independently of each other.

Note: You can enforce a specific execution order only for statically assigned test cases, with dynamically assigned ones following in a random order only after the static ones have been completed.

Steps

1. Determine where you want to reorder the test cases within your project:
   • Within a test set.
   • Within a test execution.
2. Enable the Enforce Execution Order toggle.
   • Go to Test Sets, and then to the Configuration tab, and select the Enforce Execution Order toggle.
   • Alternatively, if you want to reorder within a test execution, go to Execution, and then to the Settings tab, and enable the Enforce Execution Order toggle.
3. After you've enabled the execution order, return to the tab where you see the test cases. Depending on where you've activated the enforcement, follow these steps:
   • If you're within Test Sets, return to Test Cases and select Reorder.
   • If you're within Execution, return to Results and select Reorder.
   This opens the Reorder test cases view where you can arrange the order of test case executions.
4. Drag and drop test cases to reorder them in the test set or test execution.
5. When you finish reordering the test cases, select Save.

When you execute the test set, the test cases will run in the order you previously saved.

Re-executing test executions

If you re-execute a test execution the old execution results are deleted and replaced with the new results. If you don't want to replace a test exection result, then run the test set again. This will generate new test execution results, independent from the previous ones.

The pre-defined user roles that can perform re-executions are: Test Manager, Test Executor and Manual Tester.

Re-executing test set executions

1. Go to Execution and choose one of the following:
   • Select More Options for the test execution that you want to redo.
   • Open the test execution and select Tasks.
2. Select one of the following:
   • Re-execute manually (available only for manual test executions)
   • Re-execute automated (available only for automated test executions)
   The Re-execute dialog appears.
3. Select Confirm and execute the manual steps or wait for the automated test set to finish.
   Note:

   Re-executing test set executions doesn't display the activity coverage anymore.

Re-executing test case executions

You can re-execute one or more test case executions. For data-driven test cases, you can only re-execute one test case execution at a time. You can't re-execute data-driven test case executions in bulk.

Note: Even if you publish a newer version of the test case, the version that you re-execute is the same version that is recorded in the test execution.

1. Go to Execution and open a test set execution.
2. Select the checkboxes for one or more test case executions and then select Re-execute selection.
   
   

   

   
   
3. Select one of the following:
   • Re-execute manually (available only for manual test executions)
   • Re-execute automated (available only for automated test executions)
   The Re-execute dialog appears.
4. Select Confirm and execute the manual steps or wait for the automated test case to finish.

Troubleshooting automated executions

Automated execution could not start since the package was not found on Orchestrator. Only test cases from packages which have been published to Orchestrator can be executed.

Solution

You need to publish the Studio package to Orchestrator. This issue occurs when a test case is linked from Studio to Test Manager but the package is not yet published to Orchestrator.

Test cases could not be executed since no executable versions could be found. Only test cases published with the latest package version to Orchestrator can be executed.

Solution

This issue occurs when test cases in a test set are not published in the latest Studio package to Orchestrator. You might get the same error when there is no process/release found on Orchestrator for the Studio package.

Automated execution could not start since no automated test cases could be found in this test set.

Solution

You need to have at least one automated test case to use Execute Automated. For more information, see Conditions.

There was an error executing the test set. Please try again.

Solution

If execution of the test set fails for any reason other than the ones specified above, get help from UiPath Support Services.