Plugging the leaks: best practices for troubleshooting Java Memory Leaks

[#1]

This blog was written by Piotr Findeisen, HP Diagnostics Sr. Software Engineer

Java technology is widely used today for server side programming. One of the most prominent issues affecting deployed Java applications is the so called Java memory leak. Because Java offers automatic memory management, Java memory leaks are actually only unintentional object retention cases. However, the former term is universally used and recognized, so we also use it here.

The most important effects of the Java memory leak for long running applications are as follows:

• the performance of the application degrades over time, because of increasing frequency of the Java Virtual Machine (JVM) garbage collector (GC) runs,
• the application eventually terminates with an OutOfMemoryError.

This is a serious problem for business critical applications, as it wastes computing (and human) resources, causes unpredictable outages in the IT environment and can lead to lost business.

During this 3 part blog series we will discuss the following topics:

1. Understand Memory Leaks and the challenges faces with diagnosing and fixing Java Memory Leaks
2. Take a look at traditional troubleshooting approaches
3. Introduce a new novel approach that will change the way you troubleshoot Java Memory Leaks

Understanding Java Memory Leaks

Many managers and even some non-Java developers think that Java automatic memory management (GC) prevents the memory leaks from occurring. Without going into technicalities, let’s just say that this is a very incorrect expectation.

The nature of the problem is that the affected application, due to some faulty coding, holds references to more and more objects over time, so that the GC cannot reclaim the memory occupied by the objects. This results in the heap growing larger over time, which eventually leads to aforementioned crash.

Neither attempts to increase the heap size, nor enforcement of additional GC cycles fix the problem. They may just delay the deadly effect of the memory leak at the price of wasted resources.

The Challenge

The difficulties in diagnosing and fixing Java memory leaks stem from the following factors:

• The location of code (stack trace) provided by the JVM when it runs out of memory has nothing to do with the location of the faulty logic in the code.
• Using multiple frameworks, different vendors, several open source components, with additional home-grown glue code in contemporary Java applications is a norm rather than an exception. As a result, there is not a single developer or even a team that is responsible for, or can understand all the details of the whole application. It is not uncommon that potentially involved parties try to put the blame on others, claiming that their components have been thoroughly tested and are memory-leak free. What is interesting, their claims can be often justified, as it might require multiple components working together for the memory leak to occur.
• Some of the leaks escape detection during application development and even stress testing and appear only in applications deployed in production environment. At the same time, the leaking issue is often hard to reproduce in the test environment due to the fact that most test environments do not match their production counterparts in scale and the real-world load is always different than the test loads.

The analysis of the problem leads to the following requirements for a tool for diagnosing Java memory leaks in production environment:

• It must detect the leaks proactively, well before the application crashes. This is to allow the IT personnel to come up with emergency actions, such as procuring backup servers or schedule a controlled restart, to minimize the business impact of the incident.
• It must identify the leaking data structure (usually a collection object), and, more importantly, the faulty code that is responsible for the leak. This is because the ultimate goal is to fix the memory leak, and this can be done only by fixing the code.
• It must scale up to handle contemporary server side applications (multi-gigabyte heaps).
• It must not affect the application availability and performance (ultra low overhead).

The next blog in this 3 part series will go over some of the traditional troubleshooting approaches for Java Memory Leaks.

This blog was written by Piotr Findeisen, HP Diagnostics Sr. Software Engineer