TomcatExpert

Setting Up Measurement of Garbage Collection in Apache Tomcat

posted by dmikusa on November 16, 2011 08:15 AM

Have you ever seen this scenario before? A user has deployed an application to a Tomcat server. The application works great during testing and QA; however, when the user moves the application into production, the load increases and Tomcat stops handling requests. At first this happens occasionally and for only 5 or 10 seconds per occurrence. It's such a small issue, the user might not even notice or, if noticed, may choose to just ignore the problem. After all, it's only 5 or 10 seconds and it's not happening very often. Unfortunately for the user, as the application continues to run the problem continues to occur and with a greater frequency; possibly until the Tomcat server just stops responding to requests all together.

There is a good chance that at some point in your career, you or someone you know has faced this issue. While there are multiple possible causes to this problem like blocked threads, too much load on the server, or even application specific problems, the one cause of this problem that I see over and over is excessive garbage collection.

As an application runs it creates objects. As it continues to run, many of these objects are no longer needed. In Java, the unused objects remain in memory until a garbage collection occurs and frees up the memory used by the objects. In most cases, these garbage collections run very quickly, but occasionally the garbage collector will need to run a “full” collection. When a full collection is run, not only does it take a considerable amount of time, but the entire JVM has to be paused while the collector runs. It is this “stop-the-world” behavior that causes Tomcat to fail to respond to a request.

Fortunately, there are some strategies which can be employed to mitigate the affects of garbage collections; but first, a quick discussion about performance tuning.

Performance Tuning Basics

The first rule that you need to know: measure, adjust, and measure again. Simply put, measure the performance of your Tomcat instance before you make a change, make one change and then measure the performance of Tomcat after you have made the change. If you follow this pattern, you will always know exactly how the change you made affects the performance of your Tomcat instance.

One tip, which will help to make your measurements more accurate, is to run a load generating tool like JMeter or Selenium on the web applications deployed to your Tomcat instance. Because many garbage collection issues only occur under load, it's important generate some load on your system while you are taking measurements. Not only will this help you to more accurately replicate and test garbage collection issues, but it will also help you to minimize potential problems when you migrate your new garbage collection settings into production.

Now for the second rule: do not blindly apply configuration settings to your Tomcat instance. Just because a setting is mentioned in this or any other article does not mean that you can apply it to your configuration without first applying rule number one. It is very important to realize that your Tomcat instance is likely hosting a unique set of applications which in turn have a unique set of memory usage patterns. Because of this, the configuration settings that make one Tomcat instance run blazingly fast could easily cause crippling performance issues for another. What's worse, in practice you won't see such a dramatic change in performance. Instead, what will likely happen is that the performance will change by a smaller, less noticeable amount. So unless you've measured the performance before and after, you may not even know that you've just hurt the performance of your Tomcat instance. Remember: measure, adjust and measure again. Always. No exceptions.

Measuring Performance

Now that we all know that we need to measure the performance of our Tomcat instances prior to making any changes, that brings up the question, how do we measure the performance? It depends on what is important to your application. For some applications, individual response time may be important, while others will value throughput (i.e. how many requests Tomcat can process over some interval). For the purposes of this article, we are going to to look at something more specific to the JVM, garbage collection performance.

Garbage collection performance is a good metric to use both because it can heavily impact things like response time and response throughput and because it's easy to measure, even in a production system. To measure the performance of garbage collection we simply enable garbage collection logging.

This is done by adding the following JVM options to the CATALINA_OPTS variable in the bin/ setenv.sh|bat file for your Tomcat instance.

  • -Xloggc:$CATALINA_HOME/logs/gc.log or   Xloggc:%CATALINA_HOME%/logs/gc.log
  • -XX:+PrintHeapAtGC
  • -XX:+PrintGCDetails
  • -XX:+PrintGCTimeStamps
  • -XX:-HeapDumpOnOutOfMemoryError

Once these options are added and Tomcat is restarted, you should see a file logs/gc.log which will contain logging statements from the garbage collector. In that log file, you should see lines like the following:

1.612: [GC [PSYoungGen: 12998K->1568K(18496K)] 12998K->1568K(60864K), 0.0054130 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]

These indicate at what point in time the garbage collection occurred, the memory statistics for the heap at the point in time when garbage collection ran, and how much time it took for the garbage collection to run. Alternatively, you could see a line like this:

1.617: [Full GC (System) [PSYoungGen: 1568K->0K(18496K)] [PSOldGen: 0K->1483K(42368K)] 1568K->1483K(60864K) [PSPermGen: 9458K->9458K(21248K)],0.0294590 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]

This line has essentially the same information, however it is indicating that a full garbage collection has taken place (i.e. “Full GC” rather than just “GC”) and because of this it is showing memory statistics for the old generation and the permanent generation, in addition to the young generation which is normally displayed.

From this log we can now track how often garbage collections are occurring, how often full garbage collections are occurring and the execution time of a garbage collection. It is with these metrics in mind that we try to tune our Tomcat instance. Continue to modify your configuration and remeasure your performance over and over again until you have reached the goal of having both a minimal number of garbage collections and a minimal amount of time spent on garbage collections.

In the next post, we'll talk further about performance tuning the JVM, including a discussion on some of the JVM options that are commonly used and a discussion on the different collectors which are available.

Daniel Mikusa is a Software Support Engineer at Pivotal.  Daniel has been a user of Apache Tomcat for the past six years and is currently focused on providing enterprise-class support for Pivotal's Tomcat and tc Server customers. Daniel attended college at Kent State University and holds a bachelor's degree in Computer Science.

Comments

human readable timestamps in GC logs

Starting from Java 1.6.0.04 (released more than 3 years ago) you can use more convinient
-XX:+PrintGCDateStamps instead of
-XX:+PrintGCTimeStamps for human readable GC timestamps

It is rather very good,

It is rather very good, nevertheless glance at the data with this handle. mobile repairing training

Find the best essays on

Find the best essays on is my friend's profile page. famous cake bakeries

It is perfect time to make

It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you few interesting things or tips. Perhaps you could write next articles referring to this article. I want to read more things about it! Locksmith Orlando

I like to recommend

I like to recommend exclusively fine plus efficient information and facts, hence notice it: Sri Lanka Pet Passport

Really nice and interesting

Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing. bet188

It’s very informative and you

It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Mangeoire oiseau

Amazing knowledge and I like

Amazing knowledge and I like to share this kind of information with my friends and hope they like it they why I do..alin blog

Why do only so much written

Why do only so much written on this subject? Here you see more. travel umroh 2016

Its a great pleasure reading

Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that "The content of your post is awesome" Great work.Seo Uzmanı

I am always searching online

I am always searching online for articles that can help me. There is obviously a lot to know about this. I think you made some good points in Features also. Keep working, great job ! Vantage Point Title

I am impressed by the

I am impressed by the information that you have on this blog. It shows how well you understand this subject. Escort service

This is an excellent post I

This is an excellent post I seen thanks to share it. It is really what I wanted to see hope in future you will continue for sharing such a excellent post. www.tricks5.com

It’s very informative and you

It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. no deposit casino bonus codes

Such a very useful article.

Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. Prepping

Actually I read it yesterday

Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written. ps4 custom controller

I am very happy to read this

I am very happy to read this article .. thanks for giving us go through info.Fantastic nice. I appreciate this post. Hörlurar

Going to graduate school was

Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time.Bay Area Movers San Mateo

In this case you will begin

In this case you will begin it is important, it again produces a web site a strong significant internet site: chiropractor

Well, this got me thinking

Well, this got me thinking what other workouts are good for those of us who find ourselves on the road or have limited equipment options. circuito cerrado

A very awesome blog post. We

A very awesome blog post. We are really grateful for your blog post. combat, law enforcement You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. แบตเตอรี่รถยนต์

Welcome to the party of my

Welcome to the party of my life here you will learn everything about me. Centro de Servicio Apple

Here you will learn what is

Here you will learn what is important, it gives you a link to an interesting web page: Equipo Contra incendio

The matter you we highlighted

The matter you we highlighted above is surely value able anyone to workout for. the concept of your article is very true and It will result in a positive way. escort en la serena

Thanks for the nice blog. It

Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. escort en calama

I most likely appreciating

I most likely appreciating each and every bit of it. It is an incredible site and decent impart. I need to much obliged. Great employment! You all do an incredible blog, and have some extraordinary substance. Keep doing awesome. escort en rancagua

Such sites are important

Such sites are important because they provide a large dose of useful information ... you can buy UK Lingerie here

Below you will understand

Below you will understand what is important, the idea provides one of the links with an exciting site: buy matcha green tea

It's superior, however ,

It's superior, however , check out material at the street address. Obsession Phrases

sl786982

If you set out to make me think today; mission accomplished! I really like your writing style and how you express your ideas. Thank you. Muswell Hill Piano Teacher

sl786982

sl786982

Thank you a bunch for sharing this with all of us you actually realize what you are talking about! Bookmarked. Please also seek advice from my site =). We could have a hyperlink change contract between us! Alexander Technique Finsbury Park

sl786982

sl786982

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? psychotherapy w4

sl786982

sl786982

I high appreciate this post. It’s hard to find the good from the bad sometimes, but I think you’ve nailed it! would you mind updating your blog with more information? six pack abs after 40

sl786982

sl786982

It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing. minecraft hosting

sl786982

wow this saintly however ,I

wow this saintly however ,I love your enter plus nice pics might be part personss negative love being defrent mind total poeple , Day Trader Salary

sl786982

Taking these steps as soon as possible will help you to save money in the long run, whereas inaction can send your overall costs soaring as the damage gets much worse over time. App Store Optimization

sl786982

sl786982

I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. Thank you for the post. click here

sl786982

wertw

Watch Bad Moms Online
Watch Rings Online
Watch Ghostbusters Online
Watch Mike and Dave Need Wedding Dates Online
Watch Miracles from Heaven Online
Watch The Purge 3 Election Year Online
Watch Ratchet and Clank Online
Watch The Secret Life of Pets Online
Watch The Nice Guys Online
Watch The Jungle Book Online
Watch Alice Through the Looking Glass Online
Watch Warcraft Online
Watch The Brothers Grimsby Online
Watch The Jungle Book Online
Watch Teenage Mutant Ninja Turtles 2 Out of the Shadows Online
Watch Free State of Jones Online
Watch The Other Side of the Door Online
Watch Me Before You Online
Watch Central Intelligence Online
Watch The Secret Life of Pets Online
Watch The Nice Guys Online
Watch Captain America Civil War Online
Watch Star Trek Beyond Online
Watch Demolition Online
Watch Free State of Jones Online
Watch Now You See Me 2 Online
Watch Criminal Online
Watch Independence Day 2 Resurgence Online
Watch Star Trek Beyond Online
Watch Amityville The Reawakening Online
Watch The BFG Online
Watch Finding Dory Online
Watch Ghostbusters Online
Watch The Brothers Grimsby Online
Watch Finding Dory Online
Watch Central Intelligence Online
Watch Bad Moms Online
Watch Now You See Me 2 Online
Watch Neighbors 2 Sorority Rising Online
Watch The Huntsman Winters War Online
Watch The Conjuring 2 Online
Watch Ice Age 5 Collision Course Online
Watch Before I Wake Online
Watch The Divergent Series Allegiant Online
Watch Captain America Civil War Online
Watch Teenage Mutant Ninja Turtles 2 Out of the Shadows Online
Watch The Divergent Series Allegiant Online
Watch Jason Bourne 5 Online
Watch Midnight Special Online
Watch The Angry Birds Movie Online
Watch The Legend of Tarzan Online
Watch Jason Bourne 5 Online
Watch Money Monster Online
Watch X-Men 9 Apocalypse Online
Watch The Conjuring 2 Online
Watch The Purge 3 Election Year Online
Watch The Huntsman Winters War Online
Watch Batman v Superman Dawn of Justice Online
Watch The Angry Birds Movie Online
Watch Rings Online
Watch Batman v Superman Dawn of Justice Online
Watch Amityville The Reawakening Online
Watch The Bronze Online
Watch Criminal Online
Watch Midnight Special Online
Watch Neighbors 2 Sorority Rising Online
Watch Warcraft Online
Watch The Boss Online
Watch Independence Day 2 Resurgence Online
Watch Suicide Squad Online
Watch Demolition Online
Watch Before I Wake Online
Watch Money Monster Online
Watch X-Men 9 Apocalypse Online
Watch Mike and Dave Need Wedding Dates Online
Watch The Bronze Online
Watch Miracles from Heaven Online
Watch The Legend of Tarzan Online
Watch Ice Age 5 Collision Course Online
Watch The Other Side of the Door Online
Watch The Boss Online
Watch Suicide Squad Online
Watch The BFG Online
Watch Alice Through the Looking Glass Online
Watch Ratchet and Clank Online 213 123123 12
Watch Me Before You Online

sl786982

This content is written very well. Your use of formatting when making your points makes your observations very clear and easy to understand. Thank you. Alexander Technique Finsbury Park

sl786982

sl786982

If your looking for Online Illinois license plate sticker renewals then you have need to come to the right place. højskoler

sl786982

sl786982

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. clash of lords 2 cheat

sl786982

sl786082

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It is the old what goes around comes around routine. kostumer

sl786982

sl786982

I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often. National Open Source Software Competition

sl786982

sl786982

The post is written in very a good manner and it entails many useful information for me. I am happy to find your distinguished way of writing the post. Now you make it easy for me to understand and implement the concept. nucific bio x4 reviews

sl786982

sl786982

I needed to much obliged in your time for this sublime read!! I decidedly getting a charge out of every tad bit of it and I have you bookmarked to look at new stuff you weblog post. vfx body reviews

sl786982

sl786982

You actually make it look so easy with your performance but I find this matter to be actually something which I think I would never comprehend. It seems too complicated and extremely broad for me. I’m looking forward for your next post, I’ll try to get the hang of it! six pack shortcuts 2 reviews

sl786982

sl786982

Hi I found your site by mistake when i was searching yahoo for this acne issue, I must say your site is really helpful I also love the design, its amazing!.
joint pain relief codes jonathan bender

sl786982

sl786982

I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. business growth consulting services

sl786982

sl786982

I am very enjoyed for this blog. Its an informative topic. It help me very much to solve some problems. Its opportunity are so fantastic and working style so speedy. Halstedhus Efterskole

sl786982

sl786982

I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often. auto like app

sl786982

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.