Understanding the language of hosted load testing

I recently got an email from a colleague which was an invitation from a testing service vendor which does load testing on "The Cloud" (internet-hosted testing service).  The invitation included some misleading language about load testing that I think can be confusing to engineers that are new to performance testing.  So, here's the hook question they used in their invitation email:


     " Interested in learning how to load test your
applications from an outside-in customer perspective, so you can find and
resolve problems undetectable by traditional behind-the-firewall tools?"


Aside from the overly-casual, marketing-savvy tone of the sentence, there's actually so many hidden assumptions in this sentence it might be helpful for us to break it down:


     "Interested in learning how to load test your applications..."


Well, that's obvious...of course we are interested in learning about load testing our applications.  When this term is used generically as 'load test' I always point out that there is an assumption about the definition for load testing.  Don't be fooled by this over-simplified language - because you might be led to think that doing performance testing is simple and easy.  Like anything in IT...it's usually not simple, and often much less easy.  Also, there are many different forms of performance testing, depending on the objectives for the testing; capacity planning, scalability, configuration optimization, query tuning, migration, disaster recovery & failover and stress testing...just to mention a few. 


     "from an outside-in customer perspective..."


We know this vendor was offering testing services from OUTSIDE the firewall, generating load IN to the application under test.  This is usually a phase of testing that comes in addition your normal "in-house" performance tests, right before the system goes live, by running the load from the external network and infrastructure outside the company firewall or at a hosted facility.  But it is more important to understand the concept of "outside-in" is actually the normal definition of most kinds of testing and especially for black box test design.  To understand this, just ask yourself the inverse question: how would you conduct "inside-out" testing?  My point here is that they mention "customer perspective" which is inherently an "outside-in" perspective...because end-user customers see almost every application from some type of external interface (GUI, or CLI).  Essentially every good test is inherently designed from a customer-perspective. Customer requirements do exist even if you do not document them, or even think about them.  There is a customer (or end-user) somewhere that will be impacted by the system's behavior.  In your tests, those requirements should be directly applied in the test script and test cases themselves.


     "find and resolve problems"


Well, it would be a shame to find a problem and not resolve it.  Wouldn't you agree?  For many years now there have been complementary solutions to performance testing tools that enable profiling and diagnostics on the system under test.  It's very common now to have a performance team that includes not only testers, but developers and architects that can repair the application code and queries on-the-spot, right when a bottleneck is found.  We hear from many developers using LoadRunner for unit performance testing, and they find & fix bottlenecks so quickly...it is perceptibly a single task to them.


     "undetectable by traditional behind-the-firewall tools"


Undetectable?  Really?  There's an implication here that your performance testing environments do not include enough of the production infrastructure to find and resolve bottlenecks that would usually only exist in production.  That's only true if you don't replicate 100% of the production environment into your test lab - which is a common limitation for some companies.  But let me be very clear - that is not a limitation of the testing tool.  It is only a limitation of your resources or imagination.  To be totally honest, LoadRunner already fully supports testing and monitoring and diagnostics of nearly 100% of your production environment.  You can even run LoadRunner against your ACTUAL production systems, if you want to (although we don't recommend overloading production...in fact, please don't do that to yourself).  And don't forget, a good replacement for the actual production infrastructure is a virtualized or emulated infrastructure - using solutions like Shunra or iTKO LISA Virtualize.


The word "traditional" is just a derogatory connotation which seeks to discredit any technology that existed before today.  This usually means that there is also very little respect for the existing discipline of performance testing as it is commonly defined and conducted today.  The truth is there is nothing traditional about LoadRunner or load testing.  And to be very honest there's nothing modern about this "outside-the-firewall" testing service provider.  HP SaaS (formerly Mercury's ActiveTest) has been doing this type of testing for nearly 10 years...and they've been doing it successfully with LoadRunner, year-over-year with every new technology innovation that's come along the way.


Don't get me wrong - I do agree there are some physical bottlenecks that cannot be detected "behind-the-firewall".  Those are bottlenecks you might find with your ISP or Teleco provider in their systems or switch configurations.  Maybe the routing tables for global internet traffic aren't ideal for your application end-users in New Guinea.  Or maybe the CDN systems are having difficulty with performance throughput and simultaneous cache replication.    But if you find a bug with those OTHER COMPANIES...how do you get those bugs fixed?  Can you force them to optimize or fix the issue?  Is your only option to switch to another external provider with better performance, but perhaps other risks?


So, if we were to re-write this sentence with something more accurate, transparent and honest:


     "Are you interested in learning how to conduct effective seasonal spike testing of your production systems from outside-the-firewall, so you can enhance your existing internal performance testing efforts by diagnosing additional problems that you'll find with the external production infrastructure that you probably don't have in your own testing lab?"


(I guess it doesn't sound as catchy, eh?)


 

Comments
vikass143 | ‎06-27-2011 05:56 AM
Hi, If an application is hosted on a cloud and say we want to check the elasticity of the application (dynamic ramp up and ramp down) how can one check the performance of the application when server and added or removed on the fly based on varying user loads? Thanks, Vikas
Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Mark Tomlinson is a software tester and test engineer. His career began in 1992 with a comprehensive two-year test for a life-critical trans...


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation