How repeatable is your performance test?

Guest post by Sarah Roderus,

Vice President, Consulting Services

TCT Computing Group, Inc.

 

It doesn’t matter if you are new to LoadRunner or Performance Center or if you are a seasoned user—understanding the sequence seed is important.

 

T-image__sw__overview__loadrunner-components__560x352--C-tcm245-1246259--CT-tcm245-1237012-32.jpgWhat is the purpose behind randomizing values in a load test? What is the random sequence seed in LoadRunner/Performance Center? These are questions students often ask in LoadRunner and Performance Center training classes. I want to give you the answers to these questions today, (and you won’t have to sit through a lecture to get them).

 

People are not robots

 

When designing a performance test, a big challenge is to have one single script that behaves like a group of real users. Real users are not robots. Real users are human—who move at different speeds and select different values. For example, I walked into a retail store the other day to purchase a new pair of tennis shoes for my son. Unfortunately, they didn’t have his size and color selection in stock.

 

So, I decided to place an order online while in the store and have the shoes shipped directly to my house. Rather than having the salesman go through the painstaking effort of typing my contact information, I asked him if I could type the information and proceed online. Being a rather fast typist, (okay, I am a rather experienced online shopper as well) I completed entering all of the information and submitting the order in approximately 1.5 minutes.

While some users may take 1.0 minute to complete this task, other users may take 3 minutes, yet others may take 4.5 minutes. Randomizing the think time values is an important step to getting that one single script to behave like real people instead of robots.

 

The importance of the sequence seed

 

When you complete the think time, pacing values and parameterized values that can be randomized; there is one additional setting to consider—j the random sequence with seed setting. By setting the sequence seed before running your test and using the same sequence seed value for a second test, you are using a mathematical technique of variance reduction. When you simulate the performance of two different systems, there are basically two sources of variation between them: 

  • Differences in the performance of the two systems
  • Differences in the (randomly generated) workload 

If you can force the simulation to use the same workload for both cases, you have eliminated one of those sources by testing the performance of the two systems against the same workload. This will increase the precision of the estimates that can be obtained for a given number of iterations.

 

After executing your baseline test, a piece of code is identified to be a performance issue. The normal development cycle occurs, and the modified code ends up in your environment, ready for another round of performance testing. Setting the sequence seed before executing the test again enables you to use the internal LoadRunner capabilities, allowing you to repeat random sequences of numbers.

 

The secret of duplication

 

Beware that any modifications to the test, such as: modifying the values in the parameter file that are randomly selected, think time values, adding a new business process flow, etc. will defeat your attempt to duplicate the workload from the baseline test. In order for the variance reduction technique to be properly applied, the same sequence of events with the same randomization must be identical between the runs. If you choose not to apply a sequence seed to a performance test, a different pattern of random sequence seeds could occur—potentially resulting in a better (or worse) response time.

 

Also, when using the sequence seed and relying on its capabilities to repeat the same sequence of values, consider the potential in your simulation for system performance to influence the workload. You also need to consider the potential for system performance to influence the workload when using the sequence seed and relying on its capabilities to repeat the same sequence of values.

 

As a simple example, a higher performance system might handle a number of transactions in a fixed time test. If during the baseline test, you generate a total of 1,000 transactions, but during a subsequent test, after code enhancements, you generate a total of 2,000 transactions, only the first 50 percent of the values were reproduced during the second test. Although the same random number seed was used for both runs, the workloads the two systems were tested against were different.  Depending on the random numbers generated in the second half of the test, you may over or underestimate the impact of the performance improvement over the baseline performance.     

 

By using the power behind LoadRunner’s capabilities and setting or not setting the Scenario Run-Time Setting value of the random sequence seed as appropriate, you increase your ability to analyze and understand differences between the baseline and subsequent test results.

 

What has your experience been? I would love to know your thoughts. Feel free to share them with me and the other readers in the comments section below.

Comments
Naveen_KumarN | ‎02-11-2013 04:32 AM

Thanks for the insight information about PT.

 

Regards,

NaveenKumar N
www.QAInsights.com and www.Testifications.com

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
This account is for guest bloggers. The blog post will identify the blogger.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation