Running tests in random order

A schedule that contains only user groups and tests will run each test in a user group sequentially. By adding a random selector to a schedule, you can repeat a series of tests in random order, thus emulating the varied actions of real users.

Procedure

To add a random selector to a schedule:
  1. In the Test Navigator, browse to the schedule and double-click it.
    The schedule opens.
  2. Click the name of the schedule element to contain the random selector, and then click Add > Random Selector.
  3. In the Schedule Element Details area, add the number of iterations to loop.

    Think of yourself as a "random selector." You are blindfolded, and you have a bucket that contains 10 red balls and 10 green balls. You have a 50% chance of picking a red ball, and a 50% chance of picking a green ball. You pick a ball randomly—it is red. You then replace the ball in the bucket. Every time you pick a ball, you have a 50% chance of getting a red ball. Because the ball is replaced after each selection, the bucket always contains 10 red balls and 10 green balls. It is even possible (but unlikely) that you will pick a red ball every time.

  4. Set the weight of the random selector. The weight determines the statistical probability that a specific element will be selected.
    1. Right-click the random selector and click Add > Weighted Block.
    2. In the Weight field, type an integer.
      This integer shows the relative proportion that each test runs.

    Assume that a random selector contains two tests: Browse and Bid. You assign Browse a weight of 7 and Bid a weight of 3. Each time the loop is processed, Browse has a 70% chance of being selected, and Bid has a 30% chance of being selected.

    schedule with two scripts of different weights

Example

When a selector contains many different weights, you can mathematically determine the likelihood that a block will be executed. To do this, add the weights together and divide the weight for each block by that total.

For example, assume a selector contains six blocks set to the following weight:

  • two blocks set to a weight of 1
  • one block set to a weight of 2
  • two blocks set to a weight of 5
  • one block set to a weight of 9
The total of the weights is: 1 + 1 + 2 + 5 + 5 + 9 = 23. Therefore, the statistical likelihood of selection is:
Weight of block Likelihood of block being selected
1 (two blocks) 1/23. = 0.0435, or about 4.35% (for each block)
2 2/23 = 0.0870, or about 8.70%
5 (two blocks) 5/23 = 0.2174, or about 21.74% (for each block)
9 9/23 = 0.3913, or about 39.13%
Note that a higher weight increases the likelihood, but does not guarantee, that a block will be executed. Some variation might occur. For example, if you run a test 23 times, you cannot predict that the first and second blocks will execute exactly once, the third block exactly twice, the fourth and fifth blocks exactly five times, and the sixth block exactly nine times. However, the more times that the blocks are executed, the more accurate this prediction is.