How to select a performance load test solution?

Decisions, decisions, decisions…

Our life is full of decisions; especially when we are purchasing a new solution to improve our work.

That is how Mark Tomlinson kicked-off the "Survey of performance testing tools" Online Summit event.

 

I had the opportunity to sit in this Summit provided by the Software Test Professionals (STP) organization, over the last few days. The event was very unique because STP was able to bring 9 different vendors to present their "performance testing solutions".

 

Initially, Mark spoke about how we are always deciding what we "purchase" and our personal criteria for it which is subjective; deciding for a candy, ice cream, car, house, etc.

When we need to choose an IT solution, "it is an objective choice", Mark said.

His intro was to help people that "personal preference" doesn't really play a part in a business decision about buying a commercial performance testing tool.

 

We also had Shane Evans, the HP LoadRunner Product Manager, where he showed yesterday how easy it is to script with the newest HP patented technology TruClient.

 

Well, back to the initial presentation from Mark, he gave very good tips about selecting a performance testing solution. The following is a combination of what I learned from the Summit together with my experience.

 

As a first step, define your requirements for performance load test. Define what would be needed in the short term and long term. Within this first step, review the following items:

  • Do you have a defined process regarding performance testing?

If not, it is time to create one or review what is or isn’t working in the current process.

  • Define what organizations will be using the solution
  • Define what resources will be part of the evaluation

Define a quick RACI table, with people who will responsible, accountable, consulted and informed in the selection process.

  • What are the technical requirements?

A suggestion here would be to create a document with a matrix of your requirements and rate them based on importance.

 

CapabilityMatrixSample.PNG

 

(Sample of a comparative matrix provided by Mark Tomlinson)

 

Also, collect some information from developers to understand the development environments used, also spend time with operations to understand how the scripts can be leveraged when moving the application to production and vice versa.

Mark Tomlinson mentioned some of the information to be captured, as described below:

Protocol support, bandwidth emulation, scripting language, OS Supportability, data formats, extensibility APIs, monitoring, Automation (CI/CD).

  • What integrations will be required

E.g.: Integration with requirement management tool may be needed or you may need to have the test results stored for audit purposes in a central repository

 

  • What are your success criteria once you implement the solution?             

Then as you have the information above clear, you can start selecting vendors.

Through the process of selecting a tool, have a project with a realistic timeframe and ensure to keep up with communications to the team.

 

  • Define the criteria for the vendors to present. Try to get similar demos from vendors, so you will be able to compare apples to apples.
  • Determine phases to evaluate and reduce the number of vendors by phase.
  • Be prepared to ask questions! Mark and Scott Barber did a great job with questions to all the vendors during this event. Open and closed questions.

See some of the examples of questions vendors were asked that were mentioned by Mark:

Open Questions - expands the conversation into more elaboration
Let the vendor lead the conversation

Let the vendor share their thoughts about your needs

Examples:
- Tell me more about how this feature works
- Why is this feature important?
- And without this capability, what happens then?
- Describe a situation where this capability is used


Closed Questions - seeks specific, deterministic answers – confirmatory
Guide the conversation to what’s important

Examples:
- Does your tool support technology X or Y?
- Can I load my own data into the tool?
- What OS platforms do your tools support, windows, Linux, Unix?

 

And for sure, take time to try out the product!

 

During the event I posted the following question in many social media areas and got the respective responses that can also be helpful to you.

What are the most important features that you look for when shopping for a performance load testing tool?

“As a user I can see features such as, 24/7 tool support, self-explanatory features, as much as supported inbuilt protocols (for Load Runner), majorly - tool must be user friendly.”

“As a tool user I look for features like UI, help, technical forum, capabilities. As a manager it is important that my team is not coming to me with any sort of issues with the tool.”

“Identify your own performance requirements, provide point values to the requirements, then evaluate the tools against those requirements and see which tool has the highest point value”

 

At the end of the event, there was a quick presentation by both Mark and Scott. Scott went through 10 very good tips for the whole process of selecting a tool. My 3 favorites were:

  • Start with a Business needs analysis
  • When selecting a solution, review the vendor capabilities do deliver. Remember that the vendor will be your partner and you must trust them.
  • Easy buttons are great, but don't sacrifice extensibility for ease of use

 

The event was finalized with a discussion panel with questions from customers to all vendors.

Shane Evans wrapped up with the final comment:

"Do your homework. Research online.  Talk to people.  Learn as much as you can.  Many people before you have had similar objectives when it comes to application performance validation. Chances are, someone has already found the answer you are looking for"

 

If you are STP member you will be able to listen to the whole Event.

Here are the links to the recording:  STP Online Summit: A Survey of Performance Testing Tools

 

Now it is your turn: What are the most important features that you look for when shopping for a performance load testing tool?

 

Looking forward to hear from you!

Have a nice weekend!

Silvia Siqueira

Learn more about HP LoadRunner

 

Comments
aash mangrulkar | ‎08-26-2012 03:07 PM

S,

Good information overall, and it sounds like it was a great event for folks looking to educated on performance testing tools and selection criteria. Honestly not much new to add (as usual, Mark covers off most relevant areas), but I would reiterate points that should be emphasized to anyone looking to establish a performance testing process and/or vendor assessments in the space.

  • Not to oversimplify, but KISS is a good rule to keep in mind when establishing and implementing any process. We live in a complex enough world, the processes by which we are governed should streamline this, versus further complicate the maze that we run through to get apps tested and out the door. Establish your success criteria, and build as simple a process around delivering that as possible. Consistency and predictability (of the app and the process) is way more important than a large number of ‘checkpoints’.
  • The ‘devops’ mention was a little underplayed here, but is something that MUST be taken into consideration, especially given the traction Continuous Delivery has across Enterprise IT and the capabilities that WE are bringing to that market.
  • An oldie but a goodie is the need to look at the entire application – infrastructure and network emulation/virtualization really needs to be included to have real confidence in how an app is going to behave in the wild.
  • Most customers are NOT where they want to be, regardless of how mature they feel they are in the QA arena. Even though integrations are usually ‘nice-to-have’, they should be mentioned for two reasons: 1) it paints the ‘desired state’ vision for where things need to go and 2) HP has them! (as opposed to having to cobble things together behind the scenes.
  • Last thing to include (under Shane’s ‘do your homework’ phrase) are reference customers and marketechture. Of course we lead with this, but I am unaware of anywhere else that you’ll find the dominance that a single vendor has like HP in performance testing. This one is worth emphasizing for sure.

-aash

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
WW Sr Product Marketing Manager for HP ITPS VP of Apps & HP Load Runner
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.