Let's talk about: The Concurrency Factor

Recently I had this question come from some internal peers of mine:

“We have a customer who is being pushed to emulate a load test with larger number of Vusers ... by manipulating think time in the script. This customer is trying to run some 3000 Vusers for 300 vuser license. The argument is like this: 1 vuser takes 5 seconds of think time in between process. If they reduce the think time to 0, they are emulating to run 5 times more Vusers for the same license. If you have any input, let me know. Thanks!”

  So, yes - there is an issue here, but only with regards to the risks they are taking at getting a false positive test result. That scenario he explained is referring to what I call a "concurrency factor" for the load test design. 

  Basically, if you have a script that runs 1 real world user taking 10 minutes per iteration and you instead run it at 5 minute pacing it's a concurrency factor of 2x. And given the amount of pacing time (lowering from 10 minutes to 5 minutes) that would be reasonable and extrapolating the results mathematically is quite credible. But there is a major caveat to taking this approach for load testing.

  As the concurrency factor increases, the more difficult it is to have a credible extrapolation to the real-world conditions. In short, a higher concurrency factor is less of a real-world test - because it is running each virtual user thread at super-human speeds. It is the weakest point in the load test design, and here's why:


  If the think time or iteration pacing approaches zero, or almost none - it may show dramatically different resource consumption on the server. This can be seen easily when we describe the transaction throughput (or rate) for the load test. Take for example a 1000 trans/hour load test:

That could be 1000 virtual users running at 1 transaction in 60 minutes

  • the users log on each session, which consumes lots more memory
  • the users hold open their session for very long time, which tests the configuration for timeouts, connections, etc.
  • the users hardly consume much CPU for each session, so the context switching, paging and swapping may be much higher

That could be 1 virtual user running at 1000 transactions in 60 minutes

  • the user log on only 1 session, which consumes almost no memory and very different from the real-world
  • the user hold open their session for very, very short time, which is unrealistic hammering of session id and creation
  • the user takes each session and pushes through highest transaction rate, which hammers the CPU differently

Basically, this is a misunderstanding of the difference between real-world load testing and stress testing.

  
If a customer were to conduct testing with such high concurrency factor they may find painfully that they got a false positive from the test. In our experience, this situation is exactly what leads so many engineers down a path to being overly confident about the performance of their applications. And it leads directly (and blindly I might add) to massive performance issue for scaled-out production systems, which is pervasive architecture for the web.  They have greatly increased the risk of failure.

Labels: LoadRunner
Comments
(anon) | ‎01-22-2009 04:02 PM

I do a great deal of load testing in which I reduce think time to create more through put per VU to achieve the designed load with less VUsers.


Your point about the overhead of the application/environment handling a true number of users in such things as timeouts, connections, paging and swapping is valid, and I make these limitations clear to the technical people on the application.


Your example citing the extra load of multiple login is true only if you build your scripts to log in only once.   If each of the thousand users in your example would login in real life, then why would you build a script that logs in once and iterates on the transaction alone?


In short, every departure from actual expected conditions decreases the validity of the test.  Rarely if ever do I have the possibility of creating a complete production situation for a load test.  My goal is to document each departure and mitigate it in the test design where possible and to include it in the interpretation of the results.


(anon) | ‎03-25-2009 10:56 PM

Ok, I am looking at purchasing LoadRunner and would like to get some information on the level of Vuser licensing to procure.  We hope to simulate 2000 concurrent users during our load/stress testing.  Do I need to purchase a vUser license for 2000?

| ‎08-12-2009 08:46 PM

re: Lisa's question...if you purchased 2000 virtual user licenses and ran them just 3-times faster than the real user sessions (according to the video example)...that would simulate 6000 users on the site.

Keep in mind that Think Time is used within each session - in each action of the script, so that the user actions are simulated accurately, then add some buffer time after the session as the Iteration Pacing setting for the virtual user.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Mark Tomlinson is a software tester and test engineer. His career began in 1992 with a comprehensive two-year test for a life-critical trans...
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.