Dave Landers

This leak business turned into a huge effort. The nature of these leaks was such that the application’s ClassLoader was held (directly or indirectly) after it was supposed to be let go. The ways of holding a reference to a ClassLoader are numerous, and not always obvious.

Like I mentioned in my last post, the project started with us looking to increase iterative development performance (and therefore centered on redeploy times). We quickly discovered that redeploys were destabilizing the server: you could do about 3 or 4 redeploys before Hotspot would crash with a PermGen-related OutOfMemoryError. After that, the JVM was in a terrible state.

So we switched to JRockit, which was much better - but still would slow down after about 8 or 9 redeploys. JRockit doesn’t have the same kind of PermGen that Hotspot does (JRockit loads classes in heap, where they are treated by the gc like any other object).

It became obvious that something was leaking memory, and that was causing Hotspot to crash, and JRockit to swap. And since the leak happened with redeploys, I knew that the application’s ClassLoader would be the main leak symptom.

A leak in Java happens because some object is still referenced, and not able to be collected by the garbage collector. There is a “Path to GC Root” to that object. These paths to the object from:

• A static field in a class loaded by a parent ClassLoader (i.e. the System ClassLoader).
• Something held by a Thread stack.
• Something else the GC has decided not to get rid of (objects not yet finalized, those in a PhantomRef queue, and possibly SoftRefrerences).

Obviously we need some memory profiler tool. I first tried Optimizeit, since we had licenses available. This did help me find one leak, but it took forever (at least a day), and was terribly painful. The problem was that Optimizeit incorrectly identified any static field as a path to GC root. But most of these static fields were actually held by the application’s ClassLoader, and would be GC’ed if the ClassLoader were released. So I had a load of red herrings to wade through.

I also tried JRockit’s Memory Leak detector tool, but this too was way too painful, as it is not really aimed at this sort of problem. There was this great visual tool for viewing references, but with such a large system as ours, it also presented a load of irrelevant stuff to wade through.

On the suggestion of one of the JRockit devs, I tried YourKit, and this one nailed it. It correctly reports real paths to GC root, and has a neat “Find String” search. So now (since I know what strings to look for), finding a ClassLoader leak takes me about 5 minutes (and most of that time is spent grabbing the snapshot).

Next comes the hard part: once you find the reason for a leak, you have to figure out why it happened and how to fix it. That’ll be next.

Technorati Tags: leak, memory, portal, WebLogic