I can’t thank you enough for participating in the webinar: “Introduction to Performance testing” last week. It was very successful. We had a high attendance rate and had excellent questions!
Because the time was very limited to answer everyone’s questions, I am posting all the questions here for your review.
We want not only share these questions but we also want to get your input and continue the conversation. If you have experience and want to add to this blog, or have additional questions, please feel free to join us To watch the webinar click here: Introduction to Performance testing
Slides are attached in pdf format.
To see slides via slideshare click here
Our next webinar is March 6- Register here
Are you overwhelmed by the number of projects and releases that you are being asked to support with performance testing? Modern composite applications, Agile development, the Cloud and frequent releases are hallmarks of this current era. They also create increasing pressure to ensure application performance. How do you respond to these pressures to do more performance testing without radically increasing costs and dealing with multiple vendors?
Join Nationwide Insurance during this Web Event to learn how they increased their performance testing capacity by a factor of 10! Through a combination of consistent, standardized processes and tools to enable sharing and reuse, they were able dramatically improve their productivity and value to the organization.
- Petar Puskarich, Andrew Seling and Megan Shelton from Nationwide
- Gurmeen Aneja from HP Software
To see the Webinar schedule for Performance Testing click here.
Questions from the “Introduction to Performance Testing” webinar:
Why do we need to come back to planning from step 5?
- We move from step 5 back to the step 1 when we have new releases for a previously tested and needs to be re-tested. In this case, you can leverage data from current application behavior in production and use it as your baseline for tests.
What is the industry standard definition of Performance Testing?
- There are many opinions regarding the definition and practice of performance testing, but essentially we're talking about the overarching set of activities related to validating application performance in a pre-production environment.
What is meant by "application timeframe usability"?
- It is how the application is used during a period of time. For example, an application can have more users logging on during some periods of the day, or a particular week of the month, or particular month of the year. You may need to identify the spikes and test for that as well.
Please emphasize on the environment of AUT
- When I say "environment" it can be both: platform where the application has been developed and the platform that the application will run in production. My suggestion is to have discussions with the development and operations team to understand that. I will be posting a blog soon with more details on that.
At planning stage on understanding the system through a developer, how can one take a decision on how to implement performance testing? How does framework impact the path we choose for Performance Testing?
- The developer is just one information source; you may also need to collect additional information from additional resources. As an example, collect information from the product owner, enterprise architect and possible the operations team. The combination of the different data collect will help you to define the best approach to test the application.
What about the test environment: shared or stand alone? What about the target environment: baseline, benchmarks, conflicts with other applications on the same environment. Who is responsible for contacting other application owners?
- Very well said, all these information is important to take in consideration in the planning phase. It is ideal to have a resource collect all this information up front. In large organization you may find a performance testing "project manager" that will assign resources to collect this data. I also have seen the project manager helping on this activity and in other situations where the performance tester is the responsible for that. It all depends on how mature the organization is and resources availability.
You mentioned 5 phases. What percentage of time do you allocate to each phase? (i.e. 30, 30, 20, 10, 10)
- I would suggest 80 percent of time planning which includes the application assessment. The other 20 percent will be configuring the test based on the findings of the planning phase.
How does performance testing fit into the agile and scrum cycles?
- Performance testing in Agile needs to be part of every build, every cycle. You can’t afford to wait until the product goes live before executing some level of performance validation. Doing this could mean certain disaster. Instead, test early and test often to ensure success. Include component level (API) tests early in development when the UI is still being built. Add transport level users as systems come together and base functionality is available from end-to-end. Run regular baseline tests to check that each build doesn’t impact performance, and leverage the rapid development of TruClient scripts to handle the most complex of your business processes.
Hi, how can we performance test early in development? The UI would still be under development right?
- It is very important to start performance testing early in the development stages. It means that you can start the planning with the assessment at the same time the apps is being developed. As soon the first round of UI is completed, you will have the required information to start scripting and configure the test execution.
How to determine, how much of time and resources needed?
- This is an excellent question here is my consultant answer: It all depends. I will be writing a blog specific this topic.
How and where does Performance testing fit in the lifecycle of testing a project, including IST and UAT?
- In a typical waterfall scenario, performance testing is often left for the very end—either during or after UAT. This is a common but dangerous scenario because it can be difficult to fix performance issues so late in development, and stakeholders are quick to suggest fixing them “next release”. This often results in customer dissatisfaction and even outages. Our suggestion is to test as early and as often as possible. Start by working with developers and understand how interfaces connect in the application. These can sometimes be the earliest entry point into performance testing, and will serve as building blocks for testing later in the cycle. In short, don’t wait for the system to be fully built before you begin some level of performance testing.
If we do performance testing along with development wouldn't that mean we would be spending too much time editing the script as the information captured may change as the development progressed? I agree with early PT but not on an underdevelopment application rather an application which is undergoing functional testing
- It is very important to start performance testing early in the development stages. It means that you can start the planning with the assessment at the same time as the apps are being developed. As soon the first round of UI is completed, you will have the required information to start scripting and configure the test execution.
What’s ROI on any performance testing when it’s difficult to match test environment like any production environments?
- It is very rare for a company to have testing environments that match production, so it is up to the performance engineer to understand the differences between the test environment and production, and make educated assumptions on how those differences will affect performance. Tools help, but experience and the right set of skills help a lot here. This is also why a single test will not tell you how well your application will perform in production; only a comprehensive set of tests that includes a range of scenarios from baseline loads to peak and failure situations will ensure success of your applications.
What is the major benefit of early starting of performance testing?
- Besides of finding development issues early in the application development, it reduces the time to deliver the application earlier.
How the performance testing is useful in initial development stage on product?
- Besides of find development issues early in the application development, it reduces the time to deliver the application earlier.
How can you ensure that the application will perform well in production if you don't have the same hardware configuration on your performance test environment?
- For new applications the ideal is to try to simulate as much the production environment. For applications that are being re-rested, you can leverage monitoring results as your baseline. Another option is to leverage Service Virtualization to mimic as much as you can the production environment.
Technical Questions related to Performance testing
How about the resource consumption for true client technology?
- Because TruClient loads the entire application DOM into memory, resource consumption will be very close to that seen by actual end users. This varies largely by the application under test.
Is Load runner integrated with HP BSM monitoring suite so that we can run the Load runner scripts in BSM through BPM / EUM probe
- Yes, you can leverage scripts created in HP LoadRunner to HP Business Service Mgmt (BSM).
Is there a way we can pump up the users on the fly in the controller during a test execution— from 1000 to 2000 ?
- Yes, we can add users on the fly, during a load test run in both HP LoadRunner and HP Performance Center.
Does LoadRunner help to create machines (load) on the network to test a file transfer desktop application?
- Not sure I understand the question. We have protocols(such as web, ftp) that test file transfer timings under load. We can even emulate a TruClient or GUI session for the same (UI simulation). But I don’t know what they mean by machine creation.
Is the load generator a function with in Quality Center or ALM?
- This is one of the components included in LoadRunner and Performance Center.
Could you please talk about testing mobile apps specifically cross platform HTML5 apps? Does HP have anything around this?
- Yes, HP LoadRunner provides capabilities to execute mobile performance testing in either native or browser apps. We will have a webinar to focus on Mobile performance testing best practices in few months. Check often this blog: Performance testing webinars
Does it support client server applications? on .NET 3.5 platform
- Yes. LoadRunner can be used to test Client/Server apps as well as Web applications and headless components (i.e. Web Services, MQ, etc...)
On .NET 3.5 platform
Can LoadRunner vary parameters for mobile testing (network 3g vs. 4glte), signal strength, etc.
- Yes, for both
We are using performance center to do the performance testing, whenever we run the test we get some default report graphs. If we want to customize the report, how can we do that?
- This is a future feature that we are implementing as an Analysis Template.
Can you please talk about testing web services - REST/SOAP and JSON based? Does Load Runner support this?
- LoadRunner supports testing WS that are REST and Soap based. We support JSON based as well, some of via HP LoadRunner, some via ST scripts which can be replayed in HP LoadRunner. However, I would get Shane’s input on this as well.
What is "cause of failure" in Load testing scenario situation?
- This is a very broad question, and can have different interpretations. For example, a load test may be considered a failure if it doesn’t run to completion. If the application under test crashes (which is common when applications aren’t built to scale), or simply if SLAs are breached. It really depends on the business requirement, and the test scenario used.
Does your technology support performance testing in Silverlight 5 or 6?
- Yes, for both
What version of VuGen you are using?
- I am using HP LoadRunner 11.51 - the current version - Leveraging the TruClient Technology.
Hi, Is it possible to performance test SAP HANA application using HP 11.5 Thanks, Mano
What is the difference between transactions and Hits in terms of performance testing?
- Transactions are user defined and may include multiple steps or hits to an application. Hits are generally defined as individual request/response streams.
Hi, What language VUGEN will support? for example QTP supports VBSCRIPT
What version of HP LoadRunner are you using in this demo?
- 11.51 is the current version.
Is the Performance testing done in a QA environment or a production environment?
- Typically in a pre-Production or QA environment.
What is meant by 90 percentile value?
- This is a general statistical term.
What kind of recording mode used in VUGEN when record the scenario?
- I was using the TruClient protocol.
Can we use load runner for flash apps
We have a desktop file transfer application that will be sending multiple files from one machine to a lot of machine on a network (some in the same domain and some in different domains)
- This can be done in multiple ways. For example, you could use Java Template virtual users to create simple file transfer connections to the application, or you could record the business process of the desktop application itself using COM/DCOM, Winsock, or using the same APIs as the desktop application if those are available.
Does LoadRunner handle TN3270 Terminal Emulator recording and customization?
- Yes, using the RTE protocol
What about Bus Services that issue request XML on a specific subject? Can LoadRunner parameterize the XML inputs for multiple users? Is a measurement of the response times for each individual request & response?
- Yes, LoadRunner is able to work with XML data in a Service Bus environment.
For the LoadRunner scripts, can these be based on an existing QTP script for example? Or, are they unique to LoadRunner?
- LoadRunner can leverage QTP scripts as well. These are called GUI Virtual Users.
How I can automate Performance test using Load runner, will it affect the performance values.
- Tests can be run automatically using command line interface, and should in no way impact the performance values captured during the test. This is a common use case among mature customers who use LoadRunner as part of a Continuous Delivery lifecycle.
Why do we need to add max clients,.. why not 1 client and get the minimum performance values
- Single user testing is important, but the goal here is to test application stability and scalability under load.
Is TruClient applicable for all thick-client applications?
- TruClient is used for testing Web 2.0 and Mobile applications.
Please advise how do you decide baselines for an application
- It really depends on your business requirements and expectations for a 'normal' business period.
Can QTP scripts be integrated to simulate Load Runner scenarios?
- Yes, as GUI Virtual Users.
Does HP LoadRunner support VMView client communicating to a VDI environment using PCoIP?
- Not yet, but it is on the roadmap.
Please could you give examples how Load Runner helps in identifying performance issues and support performance tuning
- HP LoadRunner has a lot of built-in monitors that are included with the purchase. This also includes an OOTB Sitescope integration for infrastructure level monitoring. Application level monitoring can be done by integrating LoadRunner with HP Diagnostics that allows you to drill down from the user transaction level to the method level in the call stack of the application to hone in on latency and a varied list of issues.
What is the best monitoring tool?
- For Performance Testing using HP LoadRunner we recommend HP SiteScope and HP Diagnostics. SiteScope points are included with the cost of LoadRunner for testing purposes.
Can LoadRunner be used on virtual servers like VMWare?
- Yes, HP LoadRunner can run on virtual environments as VMware.
Questions related to product Licensing
Do we have to buy separate license for HP Diagnostics? Or performance center?
- HP Diagnostics is embedded with HP LoadRunner and it is licensed separately. Performance Center is the HP enterprise-class solution for performance testing that allows customers to move to a center of excellence. Customers that have a project performance testing need can start with HP LoadRunner and later own when process mature, they can upgrade to HP Performance Center. Click in the links to find out more about HP LoadRunner, HP Diagnostics and HP Performance Center.
HP Diagnostics (profiler) is part of LoadRunner suite or a separated tool?
- HP Diagnostics is embedded with HP LoadRunner and it is licensed separated. Click in the links to find out more about HP LoadRunner, HP Diagnostics.
Does the license include different protocols with LoadRunner?
- HP LoadRunner is licensed per Controller and bundle of protocols. This is helpful because you rarely will need ALL protocols. The list of protocols bundles can be found here: HP LoadRunner, HP Performance Center list of protocol bundles
In the cloud and Saas, what does SaaS mean?
- SaaS - means the HP Software as a Service. We also have HP LoadRunner in the Cloud offered by hour or day - See information on HP LoadRunner in the cloud here.
Is LoadRunner available for "personal" learning use at low or no cost?
- There is a 3-day trial license available. VuGen can be used for scripting (i.e. development/learning) beyond the 30-day period. In case your license trial expires you can contact HP for additional extension.
Is there a trial license and for how long?
- A 30-day trial is available via hp.com/go/loadrunner
Are we allowed to install LoadRunner on our system for doing the testing
- If you meant to say try LoadRunner, yes - 30 day trial is available via www.hp.com/go/loadrunner
Can we run just the HP Diagnostic outside the LoadRunner?
- Yes, you can run HP Diagnostics stand alone. See http://www.hp.com/go/diagnostics
Check our Webinar schedule for Performance testing here
Thanks again for participating!
Join our live community discussion: Huddleup for Performance testing (Tweetchat)