Running tests from Azure DevOps as a Pipeline job

You can run test assets from Azure DevOps as a Pipeline job to test the performance of your application under test.

Before you begin

You must be familiar with working with Azure DevOps.
You must have completed the following tasks:
- Been granted access to the Azure DevOps portal.
- Created an Organization in Azure DevOps.
- Created and configured the repository for the project in Azure DevOps.
- Started the Azure DevOps pipeline agent.
- Closed Rational® Performance Tester.
- Optional: Created test cases under test plans in Azure DevOps if you want to view the results of the test runs on the Test Plans dashboard.
- Optional: Created encrypted secrets or environment variables for your personal access token (PAT), if you want to use an encrypted PAT.

About this task

You must provide the values for certain fields for the selected task to run tests from Azure DevOps. The minimum fields that you need to configure along with Display name and Product Path are IMShared Path, Workspace Location, Project Name, and Test Suite Name or Configfile. The remaining fields are optional.

For example, if you provide a complete path of a configuration file that contains the parameters for a test or schedule run in the Configfile field, then you need not provide values for the IMShared Path, Workspace Location, Project Name, and Test Suite Name fields.

If you have created test cases under test plans in your Azure DevOps project, you can provide the details in the Azure DevOps URL, Test Plan Name, Test Case Name, and Azure DevOps PAT fields while you configure the job in a pipeline so that you can view the results of the test run on your Test Plan dashboard.

Procedure

Sign in to the Azure DevOps portal, if you are not already logged in.
Open the Organization page, and then select the project that you want to use to run tests.
Initialize the repository if you have created a new project by performing the following steps:
1. Click Repos from the left pane.
2. Select the Add a README checkbox if it is not selected.
3. Click Initialize from the Initialize <branch_name> with a README or gitignore section.
Click Pipelines from the left pane, and then click Create Pipeline.

Note: If your project already contains a pipeline, then you can click New Pipeline to create another pipeline.
Click Use the classic editor to create a pipeline without YAML.
Verify the values for the Team project, Repository, and Default branch for manual and scheduled builds fields, and then click Continue.
Click Empty job to add a task to the pipeline.

Note: The values for the Name, Agent pool, and Agent Specification fields are auto populated. You can change the values for the fields, if required.
Perform the following steps to add a task to the Agent job:
1. Click the Add a task to Agent job icon .
  The Add tasks pane is displayed.
2. Search for the IBM tasks defined in the IBM Rational Test Workbench extension.
  The tasks that you can select are displayed.
3. Select the IBM Rational Performance Tester Task option, and then click Add.
  
  The selected task is added to the agent job and it is displayed with a warning that some settings require attention. When you configure the details about the test run, the warning disappears.
  
  You can also remove the tasks that are not required in your job by selecting the tasks from the list. You can then right-click the tasks, and click Remove selected task(s) to remove them.

Configure IBM Rational Performance Tester Task to provide the details about the test run by referring to the following table:

The following table lists the required fields that you must provide to run the test from Azure DevOps pipeline:

Fields

Description

Configfile

Enter the complete path to a file that contains the parameters for a test or schedule run.

For example, C:\Users\IBM\rationalsdp\workspace\Project1\config.txt

For more information about creating a command line config file from Rational® Performance Tester, see the Related information section.

Display name

Enter the name for the task.

For example, testrun1

IMShared Path

Enter the complete path to the IMShared directory, if it is not in the default location.

For example, D:\Testtool\IBM\IBMIMShared

The default location of the IMShared directory is as follows:

Operating system	The default path to the directory
Windows®	C:\Program Files\IBM\IBMIMShared
Linux™	/opt/IBM/IBMIMShared
Mac	/Application/IBM/IBMIMShared

Product Path

Enter the complete path to the directory that contains eclipse.exe.

The default location of eclipse.exe is as follows:

Operating system	The default path to the directory
Windows®	C:\Program Files\IBM\SDP
Linux™	/opt/IBM/SDP
Mac	/Application/IBM/SDP

Project Name

Enter the name of the project that has performance test assets.

For example, testproj1

Test Suite Name

Enter the file name of the performance test assets relative to the project.

For example, test1.testsuite

Workspace Location

Enter the complete path of the Eclipse workspace.

For example, C:/Users/IBM/rationalsdp/workspace1

The following table lists the optional fields that you can provide to run the test from Azure DevOps pipeline:

Fields	Description
Azure DevOps PAT	Use this field to enter the personal access token of the Azure server where the test project is hosted. You can also enter the variable for the encrypted PAT that you create in Azure DevOps. The token must have read and write access. You must use Azure DevOps PAT with the following fields: Azure DevOps Project URL Test Plan Name Test Case Name
Azure DevOps Project URL	Use this field to enter the URL of the test project in the Azure server in the following format: https://<host>/<orgname>/<projectName> Note: You must enter the URL of the Azure DevOps project if you have created test cases under Test Plans in your Azure project and want to view the test results on the Test Plans dashboard. You must use Azure DevOps Project URL with the following fields: Azure DevOps PAT Test Plan Name Test Case Name
Dataset Override	Use this field to replace the dataset values during a `test` or `schedule` run. You must ensure that both original and new datasets are in the same workspace and have the same column names. When you enter a value for the Dataset Override field, you must also include the path to the dataset. You must provide the values for the Dataset Override field in the following format: /project_name/ds_path/original_ds.csv:/project_name/ds_path/new_ds.csv For example, /proj1/Datasets/ds1.csv:/proj1023/Datasets/new_ds1.csv You can override multiple datasets that are saved in a different project by adding multiple paths to the dataset separated by a semicolon. For a `test` or `schedule`, the default value is the dataset specified in the test editor or schedule editor.
Duration	Use this field to change the duration of the stages in the rate schedule. For example, Stage1=10s, Stage2=3m The stage number specified must exist in the rate schedule. Note: The Duration field creates a new copy of the rate schedule that contains the specified number of duration.
Exportlog	Use this field to specify the path of the file directory to store the exported HTTP test log in a .txt format. For example, C:/Users/Documents/tests/logexp.txt
Exportstatreportlist	Use this field to specify IDs of reports that you want to export in place of the default reports. You can provide multiple report IDs separated by a comma. You can navigate to Preferences of Rational® Performance Tester (Window > Preferences > Test > Performance Test Reports > Export Reports), and then select Show Report IDs checkbox to view the report IDs. You must use the Exportstatreportlist filed along with the Exportstats or Exportstatshtml field. For example, you can provide `http` as a value in the Exportstatreportlist field to export a Performance Report.
Exportstats	Use this field to provide the complete path to a directory that you can use to store exported reports in a comma-separated values (CSV) format. For example, C:/Users/Documents/tests Note: If you do not specify a value for the Exportstatreportlist field, then the reports that you selected in the Preferences of Rational® Performance Tester (Window > Preferences > Test > Performance Test Reports > Export Reports) are exported.
Exportstatsformat	Use this field to specify a format for the report that you want to export. You must use at least one of the following options along with the Exportstats field: `simple.csv` `full.csv` `simple.json` `full.json` `csv` `json` For example, json. You can add multiple formats for the report separated by a comma. When you want to export both simple and full type of report in a `json` or `csv` format, you can specify `json` or `csv` as the format in the field. The reports are saved to the location specified in the Exportstats field. Note: The values provided in the Exportstatsformat field always take precedence over the Export Reports options set in the product preferences Window > Preferences > Test > Performance Test Reports > Export Reports).
Exportstatshtml	Use this field to provide the complete path to a directory that you can use to export web analytic results. The results are exported to the specified directory. You can analyze the results on a web browser without using Rational® Performance Tester. For example, C:/Users/Documents/Reports
History	Use this field when you want to view a record of all events that occurred during a test or schedule run. You can use any of the following options: jaeger: To send test logs to the Jaeger UI during the `test` or `schedule` run. Note: You must set the `JAEGER_AGENT_HOST` property as an environment variable by using the command line before you use the jaeger option to send test logs to the Jaeger UI. testlog: To send test logs as traditional test logs in Rational® Performance Tester during the `test` or `schedule` run. null: To send no test logs either to the Jaeger UI or Rational® Performance Tester during the `test` or `schedule` run. For example, jaeger You can add multiple options separated by a comma to send test logs during the `test` or `schedule` run to Rational® Performance Tester and the Jaeger UI. For example, jaeger,testlog For more information about how to view test logs in the Jaeger UI and Rational® Performance Tester, see the Related information section.
Labels	Use this field to add labels to test results when the test run is complete. For example, label1, label2 You can add multiple labels to a test result separated by a comma. When you run test assets, then the labels that you added are displayed on the Performance Report in Rational® Performance Tester. The Results page of Rational® Test Automation Server displays the label that you added in the Labels field for the specific test asset in the following conditions: When you set Publish result after execution as Always or Prompt in the preferences of Rational® Performance Tester (Windows > Preferences > Test > Rational Test Automation Server). When you use the Labels field along with the Publish field. Note: When you run tests by using the double quotation marks ("") for the Labels field, then the labels in the test result do not include double quotation marks. For example, if you provide the value for the Labels field as `"100" users`, then the labels in the test result display as `100 users`. To work around this problem, you must create a command-line config file, and then run the test by using the Configfile field.
Overwrite	Use this field to determine whether a result file with the same name is overwritten or not. The default value is `false`. If the value is set to `true`, the file is overwritten and retains the same file name.
Publish	Use this field to publish test results to Rational® Test Automation Server from Azure DevOps. Remember: Before you use the Publish field, you must provide the offline user token of Rational® Test Automation Server by using any of the following methods: Set the value of the RTCP_OFFLINE_TOKEN environment variable to the offline user token of Rational® Test Automation Server by using the command-line interface. Provide the offline user token of Rational® Test Automation Server in the preferences of Rational® Performance Tester (Window > Preferences > Test > Rational Test Automation Server) You must use one of the following values in the Publish field: serverURL#project.name=name_of_the_project&teamspace.name=name_of_the _teamspace serverURL#project.name=name_of_the_project&teamspace.alias=name_of_the _teamspace_alias no The no option is useful if the product preferences are set to publish the results, but you do not want to publish them. Note: You must not include the double quotation marks ("") while providing the value for the Publish field. For example, URL_OF_RationalTest_Server#project.name=test&teamspace.name=ts1 Where: `URL_OF_RationalTest_Server` is the URL of Rational® Test Automation Server. `test` is the name of the project in Rational® Test Automation Server. `ts1` is the name of the team space. Notes: If you have a project with the same name in different team spaces, then you must append either the &teamspace.name=name_of_the _teamspace or &teamspace.alias=name_of_the_teamspace_alias options. If the name of the project or team space contains a special character, then you must replace it with `%<Hexvalue_of_special_character>`. For example, if the name of the team space is `Initial Team Space`, then you must provide it as `Intial%20Team%20Space`. Where `%20` is the Hex value of `Space` character. The values provided in the Publish field always take precedence over the Results options set in the product preferences (Window > Preferences > Test > Rational Test Automation Server > Results). The Reports information section on the IBM Rational Performance Tester Task pane displays the names of the report along with its corresponding URLs in the following conditions: When you configured the URL of Rational® Test Automation Server in Preferences of Rational® Performance Tester (Window > Preferences > Test > Rational Test Automation Server). When you set Publish result after execution as Always or Prompt in the Preferences of Rational® Performance Tester (Window > Preferences > Test > Rational Test Automation Server > Results). When you used the Publish field or the Publish field along with the Publishreports field.
Publish_for	Use this field to publish test results to Rational® Test Automation Server from Azure DevOps based on the completion status of the tests. You must use the Publish_for field along with the Publish field. You can add multiple options separated by a comma. The following are the available options that you can use for the Publish_for field: ALL: You can use this option irrespective of status of the test. PASS: You can use this option to publish test results for the tests that have passed. FAIL: You can use this option to publish test results for the tests that have failed. ERROR: You can use this option to publish test results for the tests that included errors. INCONCLUSIVE: You can use this option to publish test results for the inconclusive tests. For example, FAIL,ERROR
Publishreports	Use this field to publish specific test results to Rational® Test Automation Server. The options that you can use with Publishreports are as follows: STATS TESTLOG For example, STATS You must use the Publishreports field along with the Publish field. You can prefix the value of Publishreports with ! to publish the reports except the specified one. For example, !STATS
Rate	Use this field to change the rate of the rate runner group. For example, Rate Runner Group1=1/s, 3/m Where, `Rate Runner Group1` is the name of the Rate Runner group that has two stages. The desired rate for the first stage is one iteration per second and the rate for the second stage is three iterations per minute. Notes: The name of the Rate Runner group must match with the name in the rate schedule. The Rate field creates a new copy of the rate schedule that contains the specified number of rate.
Resource Monitoring Labels Override	Use this field to perform any of the following actions: Enable the Resource Monitoring from Service option for a performance `schedule` if the Resource Monitoring from Service option is not enabled from the schedule editor in Rational® Performance Tester. Ignore Resource Monitoring sources that were set in the performance schedule and change for a label matching mode. Replace an existing set of Resource Monitoring labels that were set in the performance schedule and run the schedule with a new set of Resource Monitoring labels. If you have added a label in Rational® Test Automation Server for a Resource Monitoring source as `rm1`, then you can provide value as shown in the following example to collect data from the source: rm1 If your label for resource monitoring contains a comma (,) then you must replace the single comma with the double comma while providing the value. For example, if you added a label to a Resource Monitoring source as `rm1,test`, then you must provide the value as shown in the following example to collect data from source: rm1,,test Notes: You can use this field only when you want to run a Rate schedule or VU schedule. You can add multiple Resource Monitoring labels separated by a comma. You must add the Resource Monitoring labels to the Resource Monitoring sources on the Resource Monitoring page in the Rational® Test Automation Server project.
Results	Use this field to provide a different name to the result file. The results file is stored in the Results directory. The default name of the result file is the name of the `test` or `schedule` with a timestamp appended.
Test Case Name	Use this field to enter the name of the test case in the Azure server. You must use Test Case Name with the following fields: Azure DevOps Project URL Azure DevOps PAT Test Plan Name
Test Plan Name	Use this field to enter the name of the test plan that is associated with the test case in the Azure server. You must use Test Plan Name with the following fields: Azure DevOps Project URL Azure DevOps PAT Test Case Name
Usercomments	Use this field to add text that you want to display in the user comments row of the report. For example, test run with dataset Note: When you run tests by using the double quotation marks ("") for the Usercomments field, then the user comments row of a report does not contain double quotation marks. For example, if you provide the value for the Usercomments field as `test run with "dataset"`, then the user comments row of a report displays the value as `test run with dataset`. To work around this problem, you must create a command-line config file, and then run the test by using the Configfile field.
Users	Use this field to override the default number of virtual users in the `test` or `schedule` run. For a `schedule`, the default is the number of users specified in the schedule editor and for a `test`, the default is one user. Note: The Users field creates a new copy of the `schedule` that contains the specified number of users.
Var File	Use this field to provide a complete path to an XML file that contains the variable name and value pairs.
VM Arguments	Use this field to specify the maximum heap size for the Java process. You can add multiple VM Arguments separated by a comma.

Note: The fields in the Control Options and Output Variables sections are from Azure DevOps that you can use to configure your test run. You can accept the default values or change the values based on your requirements.

For more information about Control Options and Output Variables, see the Related information section.

Perform any one of the following steps as per your requirement:
1. Click Save & queue to save the configurations and queue the test run in the pipeline.
  The Run pipeline dialog is displayed.
2. Click Save, and then provide a comment to save the configured settings for the task.
  The task is not queued for a run. You can save the task and run the job later. You can then click Queue to run the job at a later time.
Complete the following steps to run the test:
1. Enter a comment for the test in the Save comment field.
2. Select the agent that you configured for the test from the Agent pool list.
3. Select the Agent Specification from the list.
4. Select the Branch/tag from the list.
5. Optional: Add the variables and demands for the task run from the Advanced Options pane, if required.
6. Select the Enable system diagnostics checkbox for a detailed log view.
7. Click Save and run.
  
  The test run begins as a pipeline job and displays the progress of the job run.

Results

You have run the tests by using the IBM Rational Test Workbench extension in the Azure DevOps pipeline.

What to do next

You can open the job to view the task logs from the Jobs section. You must click the IBM Rational Performance Tester Task to view a detailed log of the test run.

The Reports information section on the IBM Rational Performance Tester Task pane displays the names of the report along with its corresponding URLs if you publish test results to IBM® Rational® Test Automation Server.

If you provided the details in the Azure DevOps URL, Test Plan Name, Test Case Name, and Azure DevOps PAT fields, then you can view the results of the test run on your Test Plan dashboard.