Apache Tomcat's new memory leak prevention and detection

posted by mthomas on April 6, 2010 10:24 AM

Tomcat has long been blamed for causing memory leaks in the permanent generation (PermGen) that lead to OutOfMemoryErrors when re-loading web applications. I was discussing this at the eJUG days conference last year when I made an off-the-cuff remark that these errors were nearly always application bugs rather then Tomcat bugs. This generated a fair amount of debate that ended up with me doing some impromtu debugging of a couple of web applications provided by the audience.

The result of the debugging was that we found a number of root causes for the memory leaks that were fairly evenly split between application code, library code and JVM code. As I was coding some fixes for these issues, I realised that Tomcat could prevent a number of these memory leaks and detect even more. Not only that, in a number of cases it looked to be possible for Tomcat to actually fix the memory leak when it detected it. This lead, ultimately, to the new memory leak prevention and detection in Tomcat 6.

There are two parts to the new memory leak features. The first part handles memory leak prevention and is implemented as a life-cycle listener, the JreMemoryLeakPreventionListener. This listener prevents memory leaks by calling various parts of the Java API that are known to retain a reference to the current context class loader. If a web application is the first code to call one of these standard Java APIs, the web application class loader will be pinned in memory which will cause a memory leak. By making sure that Tomcat code makes the calls first, the memory leaks are prevented. If you want to know the gritty details of what this listener actually does, the source code is pretty well commented.

The second part of the new features handles detection. This code executes when a web application is stopped, un-deployed or reloaded. It checks, via a combination of standard API calls and some reflection tricks, for known causes of memory leaks and - where it can - fixes them. These checks are implemented in the WebappClassLoader. A nice explanation of what these checks do has been written by Sylvain Laurent on the Tomcat Wiki. As always with Tomcat, if you want the details you can look at the source code. Start with the clearReferences() method.

The memory leak detection and prevention code is still a work in progress. More detections and preventions will be added as they are identified. If you are aware of a category of memory leaks that aren't currently detected or protected against, I encourage you to bring this to the attention of the Tomcat developers via the Tomcat developer mailing list.

Mark Thomas is a Senior Software Engineer for the SpringSource Division of VMware, Inc. (NYSE: VMW). Mark has been using and developing Tomcat for over six years. He first got involved in the development of Tomcat when he needed better control over the SSL configuration than was available at the time. After fixing that first bug, he started working his way through the remaining Tomcat bugs and is still going. Along the way Mark has become a Tomcat committer and PMC member, volunteered to be the Tomcat 4 & 7 release manager, created the Tomcat security pages, become a member of the ASF and joined the Apache Security Committee. He also helps maintain the ASF's Bugzilla instances. Mark has a MEng in Electronic and Electrical Engineering from the University of Birmingham, United Kingdom.


What about a lot (10 000+) of jsp pages without reloading?

Hi Mark,

I read your article with a lot of interest and I have two questions:

First of all, I suppose you mean prevention and detection in Tomcat 7 instead of 6 ?

Secondly, I recently contacted Spring TC support about this (see my to them mail below). They told me Tomcat 7 would be better able to cope with such an amount of data. Does this new enhancements fixes our problem?

=== start mail to Spring TC Support ===
The project/website we implemented is mostly half-static data: textual content managed through a CMS by non-technical users. The content changes, but very rarely, except for some important pages like the homepage. The servers are expected to be under normal load.

The CMS publishes its contents into concrete files, which are then served by the front-end, in this case Spring TC server. The idea is that this reduces the database load, because content is only re-published when it is changed, instead of queried on every single page access.

However, since there is some dynamic data in each page (an overview of the latest news, for example), the published pages are in turn dynamic.
The choice for the technology for these front-end dynamic pages fell on JSP because it is known to scale well.

But, while it looks like JSP scale well under load, it appears not to scale well under the number of JSP pages. With JSP pages published based on templates into many separate files, each one of these files was compiled into a separate class, leading to a very, very large number of classes. (Under Tomcat running on a Sun JVM, not only does every JSP become a class, but there are tag support classes and method accessors, too.)

Not at all surprisingly, the system eats up the permanent generation heap for breakfast. Not only does the class code get loaded into PermGen, but if I'm not mistaken, all string literals in the code end up there, too. With the content in the JSP page and the JSP page compiled into a class, this means we have the entire web page in PermGen.

This problem is pretty much inherent in the system. Because Java doesn't unload classes (at least not that I've noticed or heard of) unless the class loader is destroyed (e.g. on webapp reload), every class ever loaded stays in memory forever. Since the JSP specification requires that JSP pages are turned into classes, every JSP page accessed stays in memory forever. Eventually, all JSP pages will have to find space in the server's main memory.

Combine this with a number and size of JSP pages that grows linearly with the amount of content in the site, and you'll soon hit the limits of the hardware (or even software, on 32-bit systems).

large number of jsps and permgen

Sounds familiar. Does it have anything to do with a certain OpenCMS based solution for newspapers?

We inherited maintenance of the newspaper portal that did the same thing and the quick fix was: restart app server when probe hits the mark (you can use JMX to detect the moment when you should restart JVM).

The solution is rather simple: do NOT generate static content into JSP, instead, use templating framework like freemarker.

It has nothing to do with Tomcat, it was simply lack of knowledge of the developers and architects of the original portal/CMS. Unfortunately, nobody was willing to pay for the rewrite and we decided to keep the status quo.

Two years later we lost the contract due to some other "political" reasons (the newspaper bought a company which developed it's own CMS and they decided to switch to the new system).

JreMemoryLeakPreventionListener: how-to

Hi all,
i've read code of JreMemoryLeakPreventionListener. If i would to use that what steps must i follow ? i suppose to create a jar with only that listener (named, for example _patch.jar) and put it in common library folder (apache-tomcat-6.0.18\lib), right ? Then ?
Thanks ...

memory leak

Nice article,

Did you get a chance to try the tool from appdynamics, it is super cool and easy to use, gives all info for memory leaks

Post new comment

This question is for testing whether you are a human visitor and to prevent automated spam submissions.