TomcatExpert

Setting Up Measurement of Garbage Collection in Apache Tomcat

posted by dmikusa on November 16, 2011 08:15 AM

Have you ever seen this scenario before? A user has deployed an application to a Tomcat server. The application works great during testing and QA; however, when the user moves the application into production, the load increases and Tomcat stops handling requests. At first this happens occasionally and for only 5 or 10 seconds per occurrence. It's such a small issue, the user might not even notice or, if noticed, may choose to just ignore the problem. After all, it's only 5 or 10 seconds and it's not happening very often. Unfortunately for the user, as the application continues to run the problem continues to occur and with a greater frequency; possibly until the Tomcat server just stops responding to requests all together.

There is a good chance that at some point in your career, you or someone you know has faced this issue. While there are multiple possible causes to this problem like blocked threads, too much load on the server, or even application specific problems, the one cause of this problem that I see over and over is excessive garbage collection.

As an application runs it creates objects. As it continues to run, many of these objects are no longer needed. In Java, the unused objects remain in memory until a garbage collection occurs and frees up the memory used by the objects. In most cases, these garbage collections run very quickly, but occasionally the garbage collector will need to run a “full” collection. When a full collection is run, not only does it take a considerable amount of time, but the entire JVM has to be paused while the collector runs. It is this “stop-the-world” behavior that causes Tomcat to fail to respond to a request.

Fortunately, there are some strategies which can be employed to mitigate the affects of garbage collections; but first, a quick discussion about performance tuning.

Performance Tuning Basics

The first rule that you need to know: measure, adjust, and measure again. Simply put, measure the performance of your Tomcat instance before you make a change, make one change and then measure the performance of Tomcat after you have made the change. If you follow this pattern, you will always know exactly how the change you made affects the performance of your Tomcat instance.

One tip, which will help to make your measurements more accurate, is to run a load generating tool like JMeter or Selenium on the web applications deployed to your Tomcat instance. Because many garbage collection issues only occur under load, it's important generate some load on your system while you are taking measurements. Not only will this help you to more accurately replicate and test garbage collection issues, but it will also help you to minimize potential problems when you migrate your new garbage collection settings into production.

Now for the second rule: do not blindly apply configuration settings to your Tomcat instance. Just because a setting is mentioned in this or any other article does not mean that you can apply it to your configuration without first applying rule number one. It is very important to realize that your Tomcat instance is likely hosting a unique set of applications which in turn have a unique set of memory usage patterns. Because of this, the configuration settings that make one Tomcat instance run blazingly fast could easily cause crippling performance issues for another. What's worse, in practice you won't see such a dramatic change in performance. Instead, what will likely happen is that the performance will change by a smaller, less noticeable amount. So unless you've measured the performance before and after, you may not even know that you've just hurt the performance of your Tomcat instance. Remember: measure, adjust and measure again. Always. No exceptions.

Measuring Performance

Now that we all know that we need to measure the performance of our Tomcat instances prior to making any changes, that brings up the question, how do we measure the performance? It depends on what is important to your application. For some applications, individual response time may be important, while others will value throughput (i.e. how many requests Tomcat can process over some interval). For the purposes of this article, we are going to to look at something more specific to the JVM, garbage collection performance.

Garbage collection performance is a good metric to use both because it can heavily impact things like response time and response throughput and because it's easy to measure, even in a production system. To measure the performance of garbage collection we simply enable garbage collection logging.

This is done by adding the following JVM options to the CATALINA_OPTS variable in the bin/ setenv.sh|bat file for your Tomcat instance.

  • -Xloggc:$CATALINA_HOME/logs/gc.log or   Xloggc:%CATALINA_HOME%/logs/gc.log
  • -XX:+PrintHeapAtGC
  • -XX:+PrintGCDetails
  • -XX:+PrintGCTimeStamps
  • -XX:-HeapDumpOnOutOfMemoryError

Once these options are added and Tomcat is restarted, you should see a file logs/gc.log which will contain logging statements from the garbage collector. In that log file, you should see lines like the following:

1.612: [GC [PSYoungGen: 12998K->1568K(18496K)] 12998K->1568K(60864K), 0.0054130 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]

These indicate at what point in time the garbage collection occurred, the memory statistics for the heap at the point in time when garbage collection ran, and how much time it took for the garbage collection to run. Alternatively, you could see a line like this:

1.617: [Full GC (System) [PSYoungGen: 1568K->0K(18496K)] [PSOldGen: 0K->1483K(42368K)] 1568K->1483K(60864K) [PSPermGen: 9458K->9458K(21248K)],0.0294590 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]

This line has essentially the same information, however it is indicating that a full garbage collection has taken place (i.e. “Full GC” rather than just “GC”) and because of this it is showing memory statistics for the old generation and the permanent generation, in addition to the young generation which is normally displayed.

From this log we can now track how often garbage collections are occurring, how often full garbage collections are occurring and the execution time of a garbage collection. It is with these metrics in mind that we try to tune our Tomcat instance. Continue to modify your configuration and remeasure your performance over and over again until you have reached the goal of having both a minimal number of garbage collections and a minimal amount of time spent on garbage collections.

In the next post, we'll talk further about performance tuning the JVM, including a discussion on some of the JVM options that are commonly used and a discussion on the different collectors which are available.

Daniel Mikusa is a Software Support Engineer at Pivotal.  Daniel has been a user of Apache Tomcat for the past six years and is currently focused on providing enterprise-class support for Pivotal's Tomcat and tc Server customers. Daniel attended college at Kent State University and holds a bachelor's degree in Computer Science.

Comments

human readable timestamps in GC logs

Starting from Java 1.6.0.04 (released more than 3 years ago) you can use more convinient
-XX:+PrintGCDateStamps instead of
-XX:+PrintGCTimeStamps for human readable GC timestamps

It is rather very good,

It is rather very good, nevertheless glance at the data with this handle. mobile repairing training

Find the best essays on

Find the best essays on is my friend's profile page. famous cake bakeries

It is perfect time to make

It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you few interesting things or tips. Perhaps you could write next articles referring to this article. I want to read more things about it! Locksmith Orlando

I like to recommend

I like to recommend exclusively fine plus efficient information and facts, hence notice it: Sri Lanka Pet Passport

Really nice and interesting

Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing. bet188

It’s very informative and you

It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Mangeoire oiseau

Amazing knowledge and I like

Amazing knowledge and I like to share this kind of information with my friends and hope they like it they why I do..alin blog

Why do only so much written

Why do only so much written on this subject? Here you see more. travel umroh 2016

Its a great pleasure reading

Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that "The content of your post is awesome" Great work.Seo Uzmanı

I am always searching online

I am always searching online for articles that can help me. There is obviously a lot to know about this. I think you made some good points in Features also. Keep working, great job ! Vantage Point Title

I am impressed by the

I am impressed by the information that you have on this blog. It shows how well you understand this subject. Escort service

This is an excellent post I

This is an excellent post I seen thanks to share it. It is really what I wanted to see hope in future you will continue for sharing such a excellent post. www.tricks5.com

It’s very informative and you

It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. no deposit casino bonus codes

Such a very useful article.

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. Prepping

Actually I read it yesterday

Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written. ps4 custom controller

I am very happy to read this

I am very happy to read this article .. thanks for giving us go through info.Fantastic nice. I appreciate this post. Hörlurar

Going to graduate school was

Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time.Bay Area Movers San Mateo

In this case you will begin

In this case you will begin it is important, it again produces a web site a strong significant internet site: chiropractor

Well, this got me thinking

Well, this got me thinking what other workouts are good for those of us who find ourselves on the road or have limited equipment options. circuito cerrado

A very awesome blog post. We

A very awesome blog post. We are really grateful for your blog post. combat, law enforcement You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. แบตเตอรี่รถยนต์

Welcome to the party of my

Welcome to the party of my life here you will learn everything about me. Centro de Servicio Apple

Here you will learn what is

Here you will learn what is important, it gives you a link to an interesting web page: Equipo Contra incendio

The matter you we highlighted

The matter you we highlighted above is surely value able anyone to workout for. the concept of your article is very true and It will result in a positive way. escort en la serena

Thanks for the nice blog. It

Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. escort en calama

I most likely appreciating

I most likely appreciating each and every bit of it. It is an incredible site and decent impart. I need to much obliged. Great employment! You all do an incredible blog, and have some extraordinary substance. Keep doing awesome. escort en rancagua

Such sites are important

Such sites are important because they provide a large dose of useful information ... you can buy UK Lingerie here

Below you will understand

Below you will understand what is important, the idea provides one of the links with an exciting site: buy matcha green tea

It's superior, however ,

It's superior, however , check out material at the street address. Obsession Phrases

sl786982

If you set out to make me think today; mission accomplished! I really like your writing style and how you express your ideas. Thank you. Muswell Hill Piano Teacher

sl786982

sl786982

Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! Alexander Technique Finsbury Park

sl786982

sl786982

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? psychotherapy w4

sl786982

sl786982

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? six pack abs after 40

sl786982

sl786982

It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing. minecraft hosting

sl786982

Thank you a bunch for sharing

Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! Shipping Agency Tunisia

sl786982

I like to recommend exclusively fine plus efficient information and facts, hence notice it: money interest rate

sl786982

sl786982

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? Free Video Games

sl786982

sl786982

Very good brief and this post helped me a lot. Say thank you I searching for your facts. Continuous it.. scrape high trust flow pages

sl786982

sl786982

It should be noted that whilst ordering papers for sale at paper writing service, you can get unkind attitude. In case you feel that the bureau is trying to cheat you, don't buy term paper from it. http://www.streetinsider.com/SI+Newswire/Lean+Belly+Breakthrough+Review+-+Does+It+Really+Work%3F/11615141.html

sl786982

sl786982

The worst part of it was that the software only worked intermittently and the data was not accurate. You obviously canot confront anyone about what you have discovered if the information is not right. www.cragdon.org/perfect-biotics-review-original-ingredients-revealed/

sl786982

sl786982

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It is the old what goes around comes around routine. flat belly protocol

sl786982

sl786982

Very well written information. It will be valuable to anyone who employess it, including me. Keep doing what you are doing – for sure i will check out more posts. http://www.cragdon.org/vitapulse-review/

sl786982

sl786982

It should be noted that whilst ordering papers for sale at paper writing service, you can get unkind attitude. musikefterskole

sl786982

sl786982

We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends. Hampton Bay Lighting

sl786982

adfa

Thanks for showing this awesome post APP download and click and click and click and click and click

There is a high probability

There is a high probability that at some point in your profession organic matcha green tea powder, you or someone you know has experienced this matter. While there are several possible causes to this matter like obstructed discussions, too much put on the server, or even program particular problems, the one cause of this matter that I see over and over is extreme rubbish selection.

I need to to thank you for

I need to to thank you for this excellent read!! I definitely loved every little bit of it. I've got you book marked to check out new things you post…
Mortgage Broker Calgary

I am truly pleased to

I am truly pleased to discover this website as well as do appreciate reading through helpful content articles submitted right here. The actual suggestions from the writer had been amazing, many thanks for that reveal. thomasoldhamseo

sl786982

I was excited to find this site. I need to to thank you for your time for this wonderful read!! I definitely liked every bit of it and I have you book-marked to look at new information on your web site. Royal1688

sl786982

sl786982

Acknowledges for penmanship such a worthy column, I stumbled beside your blog besides predict a handful advise. I want your tone of manuscript... San Miguel Rescue

sl786982

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.