Plugging the leaks: Blog #3 A novel approach

[#1]

This blog was written by Piotr Findeisen, HP Diagnostics Senior Software Engineer

In Diagnostics 9.0,we introduced a new feature called Collection Leak Pinpointing (CLP).   The feature specifically targets enterprise users whose applications run 24/7. Using a patented technology, it automatically detects Java standard collection objects that perpetually grow in size. No user interaction is necessary, so the usage is extremely simple. No more guess work is required from the user either - a special facility in our solution provides a stack trace for what we call a leak point. In contrast to the allocation site for the collection object, as provided by other tools, the leak point is the location in the Java code where objects are added to a continuously growing collection.

While our goal was an overhead of less than 0.5% for typical Java applications, for many test cases in our lab the overhead was not even measurable – the run-to-run variations concealed the incurred overhead.

Usage

The standard installation of the HP Diagnostics Java Agent provides Collection Leak Pinpointing by default. Just follow the installation steps as documented, and all the Java code which runs on your JVM will be covered. This includes all deployed J2EE applications, all frameworks and third party libraries used, and the application server code as well.

Leaking collections can be inspected either in the Enterprise Client or in the Java Profiler. Both offer special views to present all detected memory leaks. Upon entering the Enterprise Client UI, the total number of currently detected memory leaks is presented along with a link to the detailed Collection Leaks view, as seen in Figure 1 (at the bottom right corner).

Figure 1: 

Clicking the link navigates to the Collection Leaks view, which offers detailed information about the leaking collection objects, as seen in the picture below.

The Java Profiler, which is a lightweight tool included with HP Diagnostics, allows viewing leaking collection objects, if any are detected, immediately after connecting to a specific probe (JVM instance). It is not necessary to start profiling, because all collection objects are monitored all the time, regardless of the probe connectivity or profiling status.

Figure 2

In the Collection Leaks view, each leaking object is represented by a row in the table in the bottom part of the view. The columns of the table present the class name of the leaking collection object, the type of elements the collection contains, the probe on which the leak occurred, as well as the collection growth rate in elements per hour, the current collection size and also the maximum size of this collection object ever recorded. Individual collection objects can be selected for charting, so the growth of the collection can be observed over time. Double-clicking on a row provides the Leak Point, which is a code location which is responsible for adding elements to the leaking collection object.

Other views presenting the memory leaks are also available, so it is possible to view the leaks associated with a particular probe, for example - as seen in Figure 4 below.

While testing Collection Leak Pinpointing with applications deployed on popular application servers, such as IBM WebSphere™ or BEA/Oracle WebLogic™, we were able to find several cases of leaking collection objects not only in our test applications, but also within the application server code. Some of the leaks were severe, and had been fixed by the respective vendors by subsequent patches to their products.

However, we were also able to see some minor leaks. For example, WebLogic 10.0 includes a feature called Self Tuning Work Manager. The purpose of this feature is to dynamically tune the application server depending on changes in the volume of traffic hitting the server. As it turned out, there is a memory leak in it, as indicated by the Leak Point provided by CLP. We could make that determination without any access to the source code (see Figure 3) – just look at the third line from the bottom.

Figure 3

While running the application over a period of one month, we observed a pretty low leak rate of about 3 collection elements per hour (see Figure 4). It is not surprising, then, that as far as we know this leak had never been addressed by any patch. Indeed, if not for CLP, this leak could have never been detected, as with such a low leak rate the JVM can stay up for several months before any increase in memory consumption is noticed.

Tuning
Contemporary applications use a variety of caching and pooling techniques to provide good performance and to ensure good usage of resources. Typically, a standard Java collection class is used as the underlying mechanism in implementation of the caches, and therefore they are subject to CLP monitoring.

If the application does not have any memory leaks, the sizes of all collections will be bound. However, filling the caches can take a long time, depending on the application configuration and the application load. By default, we give each collection (including all caches used by the application) one hour to reach saturation. Any cache that takes more time to fill will be flagged as a memory leak.

Figure 4

 

 We found that this setting works well for most applications deployed in production environments, so for most of our customers no CLP tuning is required at all.

However, sometimes the default threshold of one hour can be too short for a particular case. Users who see false positives can adjust the detection threshold value according to their knowledge of their applications and expected load. But even if they fail to change it or misjudge their own application, the CLP algorithm will automatically correct its behavior, and will remove the leaking status from the collection object some time after the collection size stops growing.

Benefits
Collection Leak Pinpointing satisfies all requirements for a tool for troubleshooting Java memory leaks in production environments. Additionally, it works automatically, without any human interaction. This is a very important feature for enterprises that deploy hundreds or even thousands of JVM instances in their production environment.

A number of standard views showing memory leaks are offered by Diagnostics. One can view the leaks for an application, for probe/JVM, or for the whole enterprise. Available sorting options let the user navigate to the detected leaks quickly and efficiently. Furthermore, it is possible to set up alerting rules and receive notifications whenever memory leaks are detected.

Conclusion
HP Diagnostics is the first tool in the market to pinpoint Java memory leaks in production environments, solving more-than-a-decade-old problem. Using a revolutionary, patented technology with near-zero overhead guarantees excellent scalability of the tool with respect to the heap size as well as to the number of JVMs monitored across the whole enterprise. Very low overhead also enables us to provide the Collection Leak Pinpointing feature by default, without any additional configuration steps and without any scoping rules that would diminish its applicability.

Fully automatic operation of CLP makes it no longer necessary for the users to browse through collection objects and figure out which ones are really leaking. Application owners receive clear and actionable information about the defects in their code with the Leak Point, which is collected for every detected memory leak without any user intervention.

View the demo on Java Memory Leak pinpointing:

This blog was written by Piotr Findeisen, HP Diagnostics Senior Software Engineer

[#2]

What a great information! I am also experiencing problems regarding Java and it is closely related to this.