Running UI tests in an Azure DevOps Pipeline

After you create the tests in IBM® Rational® Functional Tester for the application that you are testing, and after you install the IBM Rational Test Workbench extension in your organization, you can run the tests in Azure DevOps pipelines.

Before you begin

You must have completed the following tasks:

Installed the IBM Rational Test Workbench extension in your organization. See Installing the IBM Rational Test Workbench extension.
Installed an agent in your pipeline. See Azure Pipelines agents.
Created test cases under test plans if you want to view results of the test runs on the Test Plans dashboard.

About this task

After you add the IBM Rational Test Workbench extension in your Azure DevOps organization, you can use an existing pipeline or create a new one to add Rational® Functional Tester test tasks. You can install an agent or use the one that you installed in your default agent pool. You can add the Rational® Functional Tester tests to your task for the agent job, configure the task, and then run the task in the Azure DevOps pipeline.

If you have created test cases under test plans in your Azure DevOps project, you can provide the details of the Azure DevOps URL, test plan, test case, and your personal access token (PAT) while you configure the test job in a pipeline so that you can view the results of the test run on your Test Plan dashboard.

Procedure

Open your Organization page in Azure DevOps and perform the following steps:
1. Click the project you want to use.
2. Initialize the repository by performing the following steps:
  1. Click Repos from the left pane.
  2. Click Initialize from the Initialize with a README or gitignore section.
    Note: Select the Add a README check box if it is not selected.
3. Click Pipelines from the left pane.
4. Click Create Pipeline.
5. Click Use the classic editor to create a pipeline without YAML.
6. Verify the project, repository, and branch for manual and scheduled builds, and then click Continue.
7. Click Empty job.
Select Pipeline and complete the following steps:
1. Change the name for the build pipeline if required.
2. Select the Agent pool for your build pipeline.
  
  You can use the agent from the default agent pool or use the one you have installed.
3. Select the Agent Specification for the agent if required.

Add a task to the agent job by completing the following steps:

Click the Add Task icon for the agent job.

The Add tasks pane is displayed.

Search for the IBM tasks defined in the IBM Rational Test Workbench extension.

The tasks that you can select are displayed.

IBM task selection in Azure DevOps

Depending on the type of test that you have created in Rational® Functional Tester, you can select the type of task.

You must use the following table to identify the task you must select:


Type of test	Task to select
AFT suites Web UI tests Compound tests	IBM® Rational® Functional Tester Task

Select the Rational Functional Tester Task option, and then click Add to add the task to the agent job.

The selected task is added to the agent job and it is displayed with a warning that some settings require attention. You must configure the settings mentioned in Step 4.

You can also remove the tasks that are not required in your job. Select the tasks in the list that you want to remove. You can then right-click the tasks, and click Remove selected task(s) to remove them.

Configure the settings by performing the following steps:

Select the task version from the list if required.

Follow the action for the Web UI task by referring to the following table:

Note: All the required fields are marked with an asterisk (*) in the UI.

Note:

You must provide the values for the following required fields:

Workspace Location
Project Name
Test Suite Name

If you include these required parameters in a configuration file and use the Configuration File field to specify the complete file path, then these values are not required.


Field	Description	Action
Display name	Displays the name of the selected task.	Enter the name of the task.
Testcase Type	The type of test to execute.	Select Web UI from the Testcase Type list.
Product Path	The fully qualified path to the Rational® Functional Tester. This path must exist on the agent computer.	Enter the complete path of Rational® Functional Tester.
Import	The complete path to the project folder that is cloned from a source control system or remote repository.	Enter the complete path to the project folder that is cloned from a source control system or remote repository. When you use this option, you can specifiy a name for the new workspace to be created to run the imported test assets.
IMShared Path	The path to the IMShared folder on your local computer.	Enter the complete path to the location of the IBMIMShared folder. For example, C:\Program Files\IBM\IBMIMShared
Workspace Location	The complete path to the Eclipse workspace.	Enter the complete path of the Eclipse workspace.
Project Name	The name of the project containing the test.	Enter the name of the project containing the test.
Test Suite Name	The name of a test within the project to use. A test can be a Web UI test, Performance schedule, compound test or AFT suites.	Enter the name of the test that you want to run.
VM Arguments	Java™ virtual machine arguments to pass in.	Enter the Java™ virtual machine arguments. Note: You can add multiple virtual machine arguments files separated by a `comma`.
Labels	The option to add labels to the test results when the test run is complete.	Enter the labels to be added to the test results when the test run is complete. You can add multiple labels to a test result separated by a comma. For example, label1, label2 When you run test assets then the same labels are displayed on the UI Test Statistical Report in Rational® Functional Tester. If you have set Publish result after execution as Always or Prompt in the Rational® Functional Tester preferences (Window > Preferences > Test > Rational Test Automation Server) and use the Labels option, then the Results page of Rational® Test Automation Server displays the same label for the specific test asset. Note: When you run tests by using the double quotation marks ("") for the Labels field, then the labels in the test result do not include the double quotation marks. To work around this problem, you must create a command-line config file, and then run the test by using the Configuration File field. When you use the Configuration File field to run tests, then labels provided in the configuration file take precedence over the labels provided in the Labels field.
Var File	The complete path to the XML file that contains the variable name and value pairs.	Enter the complete path to the location of the variable file.
Dataset Override	For a test, the default value is the dataset specified in the test editor.	Use the `Dataset Override` option to replace dataset values during a test run. If a test is associated with a dataset, you can replace the dataset at run time while initiating the run from the command line. Note: You must use the `Dataset Override` option to replace the dataset values during a test run. You must ensure that both original and new datasets are in the same workspace and have the same column names. You must also include the path to the dataset. For example, `/project_name/ds_path/ds_filename.csv:/project_name/ds_path/new_ds_filename.csv`. You can swap multiple datasets that are saved in a different project by adding multiple paths to the dataset separated by a `semicolon (;)`. For example, `/project_name1/ds_path/ds_filename.csv:/project_name1/ds_path/new_ds_filename.csv; /project_name2/ds_path/ds_filename.csv:/project_name2/ds_path/new_ds_filename.csv`
Configuration File	The complete path to a file that contains the parameters for a test run.	Enter the complete path of the file that contains the parameters for a test run.
Results	The name of the results file. The default name of the result file is the test name with a time stamp appended.	Enter a folder name that is relative to the project to store the test results. For example, -results folder/resultname.
Overwrite	Determines whether a results file with the same name is overwritten. The default value `false` indicates that the new results file is created.	Set the value as `true` to overwrite the existing file and retain the same file name.
Export Stats	The complete path to export the report to the format specified in the Export stats format field.	Enter the values separated by comma to export those values to the CSV format.
Export stat report list	The option to list the reports that you want to export in place of the default reports, or the reports that are selected under Preferences.	Enter the list of report IDs separated by comma.
Export stats html	The option to view and analyze the results on a web browser without using the test workbench.	You can provide the complete path to a directory to export web analytic results. If you run multiple tests, do not provide a value in this field.
User Comments	The option to add and display your comments in the report.	Enter the text within double quotation mark to display it in the User Comments row of the report. Note: When you run tests by using the double quotation marks ("") for the User Comments field, then the User Comments row of a report does not contain double quotation marks. To work around this problem, you must create a command-line config file, and then run the test by using the Configuration File field.
Protocol Input	The option to run a Web UI test in parallel on different browsers.	Enter the browsers on which you want to run the Web UI test in parallel.
Export Report	The option to export the unified report to other file formats such as PDF, XML, and HTML.	Enter the following details in the text field: type=<report type>;format=<file type>;folder=<destination folder path>;filename=<name of the exported file>
Export stats format	The option to specify a format for the report that you want to export.	Specify a format for the report that you want to export. You must use the Export stats parameter along with the Export stats format parameter. You must use at least one of the following formats: `simple.csv` `full.csv` `simple.json` `full.json` `csv` `json` For example, `Export stats = <local_dir_path> Export stats format = json` You can add multiple formats for the report separated by a comma (,). If you want to export both the simple and full reports in a `json` or `csv` format, you can specify `json` or `csv` as the format in the field. The reports are saved to the location specified in the Export stats field.
publish	The option to publish test results to Rational® Test Automation Server.	Specify the server URL and project name to publish test results to Rational® Test Automation Server. You must provide the URL and offline user token of the server in Window > Preferences > Test > Rational Test Automation Server of Rational® Functional Tester before you use the publish parameter in the test script. Use the following arguments with the publish parameter: To specify the project name, use any of the following formats: serverURL #project.name=projectName&teamspace.name=name_of_the _teamspace serverURL #project.name=projectName&teamspace.alias=name_of_the _teamspace_alias You must consider the following points while providing the values: If the project name is not specified, then the value of the Project Name parameter is used. If you have a project with the same name in different team spaces, then you can append either the &teamspace.name=`name_of_the _teamspace` or &teamspace.alias==`name_of_the_teamspace_alias` options. For example: `Workspace Location = C:/Users/IBM/rationalsdp/workspace1 Project Name = proj1 Test Suite Name = Tests/testHttp.testsuite publish = https://localhost:5443#project.name=test&teamspace.name=ts1` Where: https://localhost:5443 is the URL of the server. test is the name of the project. ts1 is the name of the team space. If the name of the project or team space contains a special character, then you must replace it with `%<Hex_value_of_special_character>`. For example, if the name of the team space is `Initial Team Space`, then you must provide it as `Intial%20Team%20Space`. Where, %20 is the hexadecimal value of the space character. You must not include the double quotation marks ("") while providing the value for the Publish field. To avoid publishing of reports, use no. You can use the no option if you do not want to publish test results after the run. This option is useful if the product preferences are set to publish the results, but you do not want to publish them. For example: `Workspace Location = C:/Users/IBM/rationalsdp/workspace1 Project Name = proj1 Test Suite Name = Tests/testHttp.testsuite publish = no` If you do not use the Config File field to run the tests, then the values provided in the Publish field always take precedence over the Results options set in the product preferences (Window > Preferences > Test > Rational Test Automation Server > Results). The Reports information section on the Output window displays the names of the report along with its corresponding URLs in the following conditions: When you configured the URL of Rational® Test Automation Server in preferences of Rational® Functional Tester (Window > Preferences > Test > Rational Test Automation Server. When you set Publish result after execution as Always or Prompt in the preferences of Rational® Functional Tester (Window > Preferences > Test > Rational Test Automation Server > Results).
publish_for	The option to publish test results to Rational® Test Automation Server based on the completion status of the tests.	Specify the option to publish test results to Rational® Test Automation Server based on the completion status of the tests. You must use the publish_for parameter along with the publish parameter. The following are the available options that you can use for the publish_for parameter: ALL - This is the default option. You can use this option to publish test results for any text execution verdict. PASS - You can use this option to publish test results for the tests that have passed. FAIL - You can use this option to publish test results for the tests that have failed. ERROR - You can use this option to publish test results for the tests that included errors. INCONCLUSIVE - You can use this option to publish test results for the inconclusive tests. You can add multiple parameters separated by a comma. For example: `Workspace Location = C:/Users/IBM/rationalsdp/workspace1 Project Name = proj1 Test Suite Name = Tests/testHttp.testsuite publish = https://localhost:5443#project.name=test&teamspace.name=ts1 publish_for = FAIL,ERROR`
publishreports	The option to publish test results in Rational® Test Automation Server.	Specify the option to publish test results in Rational® Test Automation Server. The values that you can use with publishreports are as follows: FT - This is an identifier for Functional Test Report. You can use this value to publish the unified report if it is available for the selected test. See Unified reports. STATS - This is an identifier for Statistics Report. You can use this value to publish the web analytics report if it is available for the selected test. See UI Test Statistical report. TESTLOG - This is an identifier for Test Log. You can use this value to publish the test log if it is available for the selected test. See Logs overview. You must use the publishreports parameter along with the publish parameter. For example: `Workspace Location = C:/Users/IBM/rationalsdp/workspace1 Project Name = proj1 Test Suite Name = Tests/testHttp.testsuite publish = https://localhost:5443#project.name=test&teamspace.name=ts1 publishreports = STATS, TESTLOG` The values specified here override the values selected in Window > Preferences > Test > Rational Test Automation Server > Results of Rational® Functional Tester. You can prefix the value with ! to publish the reports except for the specified one in the test script. For example, `Workspace Location = C:/Users/IBM/rationalsdp/workspace1 Project Name = proj1 Test Suite Name = Tests/testHttp.testsuite publish = https://localhost:5443#project.name=test&teamspace.name=ts1 publishreports = !TESTLOG` All the reports except the TESTLOG report is published to Rational® Test Automation Server after executing the command.
Azure DevOps Project URL	The URL of the test project in the organization on the Azure DevOps server. Note: You must enter the details for this option if you have created test cases under Test plans in your Azure project and want to view the test results on the Test Plans Dashboard.	Enter the URL of the test project in the organization on the Azure DevOps server in the following format: https://<host>/<orgname>/<projectName> You must use this option with the following options: Azure DevOps PAT Test Plan Name Test Case Name
Azure DevOps PAT	Your personal access token (PAT) of the Azure server where the test project that contains the test plan is hosted. You can also add your token to a secret variable and specify the variable name in the following form: `$(variable_name)` Note: The token must have read and write access.	Enter your personal access token. You must use this option with the following options: Azure DevOps Project URL Test Plan Name Test Case Name
Test Plan Name	The name of the test plan in the Azure server.	Enter the name of the test plan. You must use this option with the following options: Azure DevOps Project URL Azure DevOps PAT Test Case Name
Test Case Name	The name of the test case in the Azure server.	Enter the name of the test case. You must use this option with the following options: Azure DevOps Project URL Azure DevOps PAT Test Plan Name

Expand Control Options and configure the settings for your task if required.
Expand Output Variables and configure the settings for your task if required.

Select the following options:
1. Click Save to save the configured settings for the task.
  
  Note: The task is not queued for a run.
  
  You can save the task to a build pipeline and opt to run the build at a later time.
2. Click Save & queue to save the configurations and queue the run in the pipeline.
  
  The Run pipeline dialog box is displayed.
Complete the following steps:
1. Enter a comment for the test in the Save comment field.
2. Select the agent that you configured for the test from the Agent pool list.
3. Select the agent specification from the Agent Specification list for the agent if required.
4. Select the branch from the Branch/tag list.
5. Add the variables and demands for the task run from the Advanced Options pane if required.
6. Select the Enable system diagnostics check box for a detailed log view.
7. Click Save and run.
  
  The pipeline summary page displays the progress of the job run.

Results

You have run the tests for the application you are testing, in the Azure DevOps pipeline.

What to do next

You can open the job to view the task logs from the pipeline summary page.
You can click Test Plans to view the test result if you specified the values for Azure DevOps Project URL, Azure DevOps PAT, Test Plan Name, and Test Case Name.
You must click the task to open the Task page to view the test results.
You can access the report URLs to view the test execution information at any point of time. The report URLs are the IBM® Rational® Test Automation Server URLs where the reports are stored.
In Rational® Functional Tester, if the Rational® Test Automation Server URL is configured in Window > Preferences > Test > Rational Test Automation Server and Publish result after execution is set as Always in Window > Preferences > Test > Rational Test Automation Server > Results, then the logs in the Task page also displays the names of the published report along with its corresponding URLs.