Decisions, decisions, decisions…
Our life is full of decisions; especially when we are purchasing a new solution to improve our work.
That is how Mark Tomlinson kicked-off the "Survey of performance testing tools" Online Summit event.
I had the opportunity to sit in this Summit provided by the Software Test Professionals (STP) organization, over the last few days. The event was very unique because STP was able to bring 9 different vendors to present their "performance testing solutions".
Initially, Mark spoke about how we are always deciding what we "purchase" and our personal criteria for it which is subjective; deciding for a candy, ice cream, car, house, etc.
When we need to choose an IT solution, "it is an objective choice", Mark said.
His intro was to help people that "personal preference" doesn't really play a part in a business decision about buying a commercial performance testing tool.
Well, back to the initial presentation from Mark, he gave very good tips about selecting a performance testing solution. The following is a combination of what I learned from the Summit together with my experience.
As a first step, define your requirements for performance load test. Define what would be needed in the short term and long term. Within this first step, review the following items:
- Do you have a defined process regarding performance testing?
If not, it is time to create one or review what is or isn’t working in the current process.
- Define what organizations will be using the solution
- Define what resources will be part of the evaluation
Define a quick RACI table, with people who will responsible, accountable, consulted and informed in the selection process.
- What are the technical requirements?
A suggestion here would be to create a document with a matrix of your requirements and rate them based on importance.
(Sample of a comparative matrix provided by Mark Tomlinson)
Also, collect some information from developers to understand the development environments used, also spend time with operations to understand how the scripts can be leveraged when moving the application to production and vice versa.
Mark Tomlinson mentioned some of the information to be captured, as described below:
Protocol support, bandwidth emulation, scripting language, OS Supportability, data formats, extensibility APIs, monitoring, Automation (CI/CD).
- What integrations will be required
E.g.: Integration with requirement management tool may be needed or you may need to have the test results stored for audit purposes in a central repository
- What are your success criteria once you implement the solution?
Then as you have the information above clear, you can start selecting vendors.
Through the process of selecting a tool, have a project with a realistic timeframe and ensure to keep up with communications to the team.
- Define the criteria for the vendors to present. Try to get similar demos from vendors, so you will be able to compare apples to apples.
- Determine phases to evaluate and reduce the number of vendors by phase.
- Be prepared to ask questions! Mark and Scott Barber did a great job with questions to all the vendors during this event. Open and closed questions.
See some of the examples of questions vendors were asked that were mentioned by Mark:
Open Questions - expands the conversation into more elaboration
Let the vendor lead the conversation
Let the vendor share their thoughts about your needs
- Tell me more about how this feature works
- Why is this feature important?
- And without this capability, what happens then?
- Describe a situation where this capability is used
Closed Questions - seeks specific, deterministic answers – confirmatory
Guide the conversation to what’s important
- Does your tool support technology X or Y?
- Can I load my own data into the tool?
- What OS platforms do your tools support, windows, Linux, Unix?
And for sure, take time to try out the product!
During the event I posted the following question in many social media areas and got the respective responses that can also be helpful to you.
What are the most important features that you look for when shopping for a performance load testing tool?
“As a user I can see features such as, 24/7 tool support, self-explanatory features, as much as supported inbuilt protocols (for Load Runner), majorly - tool must be user friendly.”
“As a tool user I look for features like UI, help, technical forum, capabilities. As a manager it is important that my team is not coming to me with any sort of issues with the tool.”
“Identify your own performance requirements, provide point values to the requirements, then evaluate the tools against those requirements and see which tool has the highest point value”
At the end of the event, there was a quick presentation by both Mark and Scott. Scott went through 10 very good tips for the whole process of selecting a tool. My 3 favorites were:
- Start with a Business needs analysis
- When selecting a solution, review the vendor capabilities do deliver. Remember that the vendor will be your partner and you must trust them.
- Easy buttons are great, but don't sacrifice extensibility for ease of use
The event was finalized with a discussion panel with questions from customers to all vendors.
Shane Evans wrapped up with the final comment:
"Do your homework. Research online. Talk to people. Learn as much as you can. Many people before you have had similar objectives when it comes to application performance validation. Chances are, someone has already found the answer you are looking for"
If you are STP member you will be able to listen to the whole Event.
Here are the links to the recording: STP Online Summit: A Survey of Performance Testing Tools
Now it is your turn: What are the most important features that you look for when shopping for a performance load testing tool?
Looking forward to hear from you!
Have a nice weekend!