HP Diagnostics: Identifying Transaction Bottlenecks

This blog will present a real-life example of using HP LoadRunner and Diagnostics to find the method level bottleneck in a slow end user transaction during pre-production. Customers can use similar Diagnostics capability in production environment, when they are analyzing the application performance bottlenecks and root causes to poor application performance. This info can be channeled back to the development organization so that the application performance issues identified in production can be quickly addressed in a patch or new application version. This type of detail level info sharing on performance issues and collaboration to remedy them, will improve the application performance management at any customer organization.

 

Customer Challenge

The testing team at a US healthcare company uses HP Load Runner to measure transaction performance of a mission critical composite application from the end user perspective. Backend components are performance tested using LoadRunner to make Web Service and database calls. While they are successful at identifying poorly performing transactions, they still struggle with identifying the root cause of observed transaction performance issues. This lack of visibility into transaction performance comes at a high price to the organization. Developers are not able to focus their performance tuning efforts on the right components. The test team struggles with finding granular component level performance timings through the transaction flow and fails to provide actionable information to the development team to respond in an agile fashion to application issues. HP J2EE/.NET Diagnostics is designed to address these issues in the development lifecycle and the conducted a brief proof-of-concept (POC) for the customer demonstrates the LoadRunner-Diagnostics integration and key capabilities it will provide.

 

Configuring LoadRunner-Diagnostics integration

See Joe Colantonio’s blog on setting up LoadRunner-Diagnostics integration:

http://www.joecolantonio.com/2011/03/01/hp-diagnostics-how-to-install-java-agent-with-loadrunner-tom...

 

Transaction Analysis

In the image below, we can see how the LoadRunner Controller UI controls the test execution. We can see the summary (1) and over-time (2) performance of the Login and Logout transactions as they are measured from a virtual user perspective for this specific load test scenario. Note that the “Run” tab (3) is selected below.

1

In the image below, we are looking at the contents of the “Diagnostics for J2EE/.NET” tab (1) in LoadRunner Controller UI. This UI provides visibility into the live performance of all the backend application components (Servlets, JSPs, EJBs, Heap, Layers, etc) for the active load test (2). We are actually viewing the Diagnostics UI embedded inside the LoadRunner Controller UI. This UI integration is unique to LoadRunner solution. The use of any alternative diagnostics solution would require the user to switch between the LoadRunner UI and an external UI constantly and try to baseline the measurement separately. This can be very time consuming and lead to misleading analysis.

2.png

As seen above, Diagnostics presents the latency of the Login and Logout transactions (3.1) within the JVM/CLR application servers. Diagnostics also presents the slowest 5 EJBs, Servlets, Web Service Operations, etc (3.2). In the load view (3.3), Diagnostics groups method level performance into logical layers of the application. The Probes view (3.4), displays the heap usage in the application servers under test. Customers prefer this view since it provides a single pane of glass into the various key backend performance indicators of the application under test. Users can double click on any chart to drill deeper into a specific area. In the above example, we double click on the transactions chart (3.5) to drill into the detailed view for transaction performance which is shown below.

 

In the image below, we have drilled into the Synthetic Transactions view in Diagnostics. We can see all performance metrics for incoming LoadRunner transactions aggregated over the active load test. Thanks to the out-of-the-box LR-Diagnostics integration, the user does not have to manually configure these transactions in Diagnostics. They are automatically seen in Diagnostics when you start your LoadRunner test scenario with Diagnostics probes. Other solutions require changes to test scripts in order to display this information in their own UI and will make the diagnostics and analysis more complex and time consuming task.

3.png

 

The above view tells us that the login transaction is taking a significant amount of time, 9.2 seconds. We can right click the Login transaction and look at its topology, as shown below. This tells us the application servers the Login transaction flows through.

4.png

 

Next, we drill into the application server requests invoked by this LoadRunner transaction. Note that this view is filtered down to requests for a specific transaction, Login in this case (1). The table lists the individual components involved and their statistics over the active load test.

5.png

The table above tells us that a single servlet (2) is causing most of the transaction latency. And we can see that latency is tied to CPU (3). Why is this request taking so much CPU? To answer this, we drill into the Java stack of servlet invocation instances (4) to identify methods consuming CPU.

 

In the image below, we are looking at the Java stack for a Servlet instance (1). Each red bar represents a method in the stack and the width reflects the method’s duration (2).  We can see that this instance had a total latency of 20.7 s of which 13.5 s was spent in CPU time (3).

6.png

 

In the image below, we are looking at the bottom portion of the above servlet instance. It shows us that the web service engine is spending most of the instance’s time in XML parsing related to the SOAP response from the downstream TIBCO service.

7.png

 

After analyzing a number of slow servlet instances, we found the SOAP SAX parsing was the actual bottleneck of customer’s slow transactions. This was exactly the type of unambiguous performance information that the test team wanted to provide their development team.

 

When a test team provides development teams with this level of visibility into transaction bottlenecks, it typically causes a shift in the relationship between the teams. The test team becomes a trusted advisor to the development team and contributes to the agility of the development process.

 

Similar sharing of diagnostics info can be achieved between the development and operations teams. The analysis example provided is one of the testimonials how the collaboration between application development, testing and operations can improve the application lifecycle management in a customer organizations today. This collaboration can be supported by the technology solutions like LoadRunner and Diagnostics, but will require much more active approach from the customer side to enable the collaborative processes between organization teams. This may require more of an organizational process development activity, rather merely using a technology or tool to enable the collaboration.

 

By: Petri Maanonen

 

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.