TomcatExpert

Setting Up Measurement of Garbage Collection in Apache Tomcat

posted by dmikusa on November 16, 2011 08:15 AM

Have you ever seen this scenario before? A user has deployed an application to a Tomcat server. The application works great during testing and QA; however, when the user moves the application into production, the load increases and Tomcat stops handling requests. At first this happens occasionally and for only 5 or 10 seconds per occurrence. It's such a small issue, the user might not even notice or, if noticed, may choose to just ignore the problem. After all, it's only 5 or 10 seconds and it's not happening very often. Unfortunately for the user, as the application continues to run the problem continues to occur and with a greater frequency; possibly until the Tomcat server just stops responding to requests all together.

There is a good chance that at some point in your career, you or someone you know has faced this issue. While there are multiple possible causes to this problem like blocked threads, too much load on the server, or even application specific problems, the one cause of this problem that I see over and over is excessive garbage collection.

As an application runs it creates objects. As it continues to run, many of these objects are no longer needed. In Java, the unused objects remain in memory until a garbage collection occurs and frees up the memory used by the objects. In most cases, these garbage collections run very quickly, but occasionally the garbage collector will need to run a “full” collection. When a full collection is run, not only does it take a considerable amount of time, but the entire JVM has to be paused while the collector runs. It is this “stop-the-world” behavior that causes Tomcat to fail to respond to a request.

Fortunately, there are some strategies which can be employed to mitigate the affects of garbage collections; but first, a quick discussion about performance tuning.

Performance Tuning Basics

The first rule that you need to know: measure, adjust, and measure again. Simply put, measure the performance of your Tomcat instance before you make a change, make one change and then measure the performance of Tomcat after you have made the change. If you follow this pattern, you will always know exactly how the change you made affects the performance of your Tomcat instance.

One tip, which will help to make your measurements more accurate, is to run a load generating tool like JMeter or Selenium on the web applications deployed to your Tomcat instance. Because many garbage collection issues only occur under load, it's important generate some load on your system while you are taking measurements. Not only will this help you to more accurately replicate and test garbage collection issues, but it will also help you to minimize potential problems when you migrate your new garbage collection settings into production.

Now for the second rule: do not blindly apply configuration settings to your Tomcat instance. Just because a setting is mentioned in this or any other article does not mean that you can apply it to your configuration without first applying rule number one. It is very important to realize that your Tomcat instance is likely hosting a unique set of applications which in turn have a unique set of memory usage patterns. Because of this, the configuration settings that make one Tomcat instance run blazingly fast could easily cause crippling performance issues for another. What's worse, in practice you won't see such a dramatic change in performance. Instead, what will likely happen is that the performance will change by a smaller, less noticeable amount. So unless you've measured the performance before and after, you may not even know that you've just hurt the performance of your Tomcat instance. Remember: measure, adjust and measure again. Always. No exceptions.

Measuring Performance

Now that we all know that we need to measure the performance of our Tomcat instances prior to making any changes, that brings up the question, how do we measure the performance? It depends on what is important to your application. For some applications, individual response time may be important, while others will value throughput (i.e. how many requests Tomcat can process over some interval). For the purposes of this article, we are going to to look at something more specific to the JVM, garbage collection performance.

Garbage collection performance is a good metric to use both because it can heavily impact things like response time and response throughput and because it's easy to measure, even in a production system. To measure the performance of garbage collection we simply enable garbage collection logging.

This is done by adding the following JVM options to the CATALINA_OPTS variable in the bin/ setenv.sh|bat file for your Tomcat instance.

  • -Xloggc:$CATALINA_HOME/logs/gc.log or   Xloggc:%CATALINA_HOME%/logs/gc.log
  • -XX:+PrintHeapAtGC
  • -XX:+PrintGCDetails
  • -XX:+PrintGCTimeStamps
  • -XX:-HeapDumpOnOutOfMemoryError

Once these options are added and Tomcat is restarted, you should see a file logs/gc.log which will contain logging statements from the garbage collector. In that log file, you should see lines like the following:

1.612: [GC [PSYoungGen: 12998K->1568K(18496K)] 12998K->1568K(60864K), 0.0054130 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]

These indicate at what point in time the garbage collection occurred, the memory statistics for the heap at the point in time when garbage collection ran, and how much time it took for the garbage collection to run. Alternatively, you could see a line like this:

1.617: [Full GC (System) [PSYoungGen: 1568K->0K(18496K)] [PSOldGen: 0K->1483K(42368K)] 1568K->1483K(60864K) [PSPermGen: 9458K->9458K(21248K)],0.0294590 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]

This line has essentially the same information, however it is indicating that a full garbage collection has taken place (i.e. “Full GC” rather than just “GC”) and because of this it is showing memory statistics for the old generation and the permanent generation, in addition to the young generation which is normally displayed.

From this log we can now track how often garbage collections are occurring, how often full garbage collections are occurring and the execution time of a garbage collection. It is with these metrics in mind that we try to tune our Tomcat instance. Continue to modify your configuration and remeasure your performance over and over again until you have reached the goal of having both a minimal number of garbage collections and a minimal amount of time spent on garbage collections.

In the next post, we'll talk further about performance tuning the JVM, including a discussion on some of the JVM options that are commonly used and a discussion on the different collectors which are available.

Daniel Mikusa is a Software Support Engineer at Pivotal.  Daniel has been a user of Apache Tomcat for the past six years and is currently focused on providing enterprise-class support for Pivotal's Tomcat and tc Server customers. Daniel attended college at Kent State University and holds a bachelor's degree in Computer Science.

Comments

human readable timestamps in GC logs

Starting from Java 1.6.0.04 (released more than 3 years ago) you can use more convinient
-XX:+PrintGCDateStamps instead of
-XX:+PrintGCTimeStamps for human readable GC timestamps

It is rather very good,

It is rather very good, nevertheless glance at the data with this handle. mobile repairing training

Find the best essays on

Find the best essays on is my friend's profile page. famous cake bakeries

It is perfect time to make

It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you few interesting things or tips. Perhaps you could write next articles referring to this article. I want to read more things about it! Locksmith Orlando

I like to recommend

I like to recommend exclusively fine plus efficient information and facts, hence notice it: Sri Lanka Pet Passport

Really nice and interesting

Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing. bet188

It’s very informative and you

It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Mangeoire oiseau

Amazing knowledge and I like

Amazing knowledge and I like to share this kind of information with my friends and hope they like it they why I do..alin blog

-XX:+PrintGCDetails not available

Just an update on Tomcat 7.0.41, -XX:+PrintGCDetails is not available anymore. I am yet to figure out the alternatives for this. Any idea on this?

-XX:+PrintGCDetails not available

That is not a tomcat setting. It's HotSpot JVM setting. You probably have a typo in setenv.sh or perhaps the Linux alternatives mechanism has changed the JVM you were using from HotSpot to somthing else unexpected?

De toutes les maisons majeurs

De toutes les maisons majeurs de l'industrie horlogère suisse, Breguet est plus que jamais le maître incontesté de la mesure précise du temps à la plus petite fraction, grâce à 95 ans de savoir-faire accumulé dans les fréquences mécaniques élevées et ultra-haute. replique montre La liste de ses premières mondiales brevetées est sans égale:

página web

There is a good chance that at some point in your career, you or someone you know has faced this issue. While there are multiple possible causes to this problem like blocked threads, too much load on the server, or even application specific problems, the one cause of this problem that I see over and over is excessive garbage collection.
página web

Toronto Wedding Limousine

that at some point in your career, you or someone you know has faced this issue. While there are multiple possible causes to this problem like blocked threads, too much load on the server, or even application specific problems, the one cause of this problem that I see over and over is excessive garbage collection.
Toronto Wedding Limousine

http://likri.com/

Simply put, measure the performance of your Tomcat instance before you make a change, make one change and then measure the performance of Tomcat after you have made the change. If you follow this pattern, you will always know exactly how the change you made affects the performance of your Tomcat instance.
http://likri.com/

candid wedding photography Hyderabad

There is a good chance that at some point in your career, you or someone you know has faced this issue. While there are multiple possible causes to this problem like blocked threads, too much load on the server, or even application specific problems, the one cause of this problem that I see over and over is excessive garbage collection.
candid wedding photography Hyderabad

Cheap content, With thanks

Cheap content, With thanks just for stating This approach skills. Wonderfully developed page, only if all of the the blogosphere obtainable the equivalent degree content and articles simply because, websites became a more desirable space. Delight continue!
embroidery supplies

Best Coffee Makers 2015

change by a smaller, less noticeable amount. So unless you've measured the performance before and after, you may not even know that you've just hurt the performance of your Tomcat instance. Remember: measure, adjust and measure again. Always. No exceptions.
Best Coffee Makers 2015

newcastle Wedding photography

In Java, the unused objects remain in memory until a garbage collection occurs and frees up the memory used by the objects. In most cases, these garbage collections run very quickly, but occasionally the garbage collector will need to run a “full” collection.
newcastle Wedding photography

PDF Filler Reviews

Listed here you'll learn it is important, them offers the link in an helpful webpage:
PDFfiller

Home Remedy Advisory

The best article I came across a number of years, write something about it on this page.
Home remedies for backache

GAME HACK

hill climb racing hack apk I have a similar interest this is my page read everything carefully and let me know what you think. clash of clans hack online -download

Baxter

Thank you so much meant for ad an extremely terrific report! I recently found your blog post perfect for my best necessities. It contains terrific together with practical articles or blog posts. Stick to beneficial financial job!
Orlando Reputation Consulting

Venus Factor Reviews

Here you will learn what is important, it gives you a link to an interesting web page:
click here to visit the official home page

The globe will be changing fast.

The globe will be changing fast. people are also becoming altered.day by day time we are becoming more dependant on degital system.you are making me think of this really.You have a great way of sharing your thoughts obat miom tanpa operasi
obat penyakit jantung koroner
obat alami asam urat.

consultant seo

référencement Within this webpage, you'll see the page, you need to understand this data. référencement | référencement

Lock Smith

locksmith auto I use only high quality materials - you can see them at: locksmiths

Prestation SEO

référencement I really appreciate this wonderful post that you have provided for us. I assure this would be beneficial for most of the people. référencement | référencement

Digital Marketing Agency Manchester

Wow! What an eye opener this post has been for me. Very much appreciated, bookmarked, I can’t wait for more!
Digital marketing agency Manchester

Manchester Web Design Agency

Nice share of information here. I really find this information really helpful for me. Thanks a lot, dude, for putting up this vital information on the internet for everybody to use.
Manchester web design agency

SEO

référencement I am always searching online for storys that can accommodate me. There is obviously a multiple to understand about this. I feel you made few salubrious points in Attributes moreover. Detain busy, awesome career! agence référencement | référencement naturel

s download software mxf

s download software mxf converter, convert p2 mxf files from your camcorder. convert mxf files to avi mp4 mov s

Marketing 1ON 1

I understand this column. I realize You put a many of struggle to found this story. I admire your process.
backlink SEO

5K Daily Profit Club Review

In this particular article, you will see a summary, satisfy browse this post.
5K Daily Profit Club

The acceptable affair about

The acceptable affair about the Patek Philippe Swiss replica watches uk accumulating is a lot of the watches on affectation are created authoritative use of the exact aforementioned abstracts that go into the authoritative of an original. That makes assertive that this watch actively isn't in actuality a replica central faculty in the word, but an eye anchored that is appropriately able-bodied fabricated for the acumen that aboriginal itself. Several websites accommodate a answer if you buy several Swiss replica watches. The absolute could cause of this comes about because a lot of humans anyhow buy added you watch every time they appear online! Whenever you appraise on the account of amazing Patek Philippe Swiss replica watches on display, you cannot bind you to ultimately buy one - abnormally afterwards you appraise the retail price. Accepting a omega replica watches in this superior at such an affordable should absolutely be surprising, and it's aswell that agency which drives humans from globally to our website every time they seek for the best superior replica watches.

http://www.rfshoeoutlets.co.uk

Food Company Names

Extremely educational post! There is a great deal of data here that can help any business begin with a fruitful informal communication fight!
Food Company Names

Richard C. Soucy | FRANK MORLEY

continue reading this Very good topic, similar texts are I do not know if they are as good as your work out. click for source

Kirsten I. Tovar | Jean P. Harris

website here Within this webpage, you'll see the page, you need to understand this data. hop over to this web-site

Femme Promo

I would recommend my profile cool promo items is important to me, I invite you to discuss this topic.

Cash For Clothes In London

Beaver says I also have such interest, you can read my profile here:
ladies and menswear that we buy from you

web design & development

I also wrote an article on a similar subject will find it at Wordpress sites development write what you think.

Flighthub reviews

Very good topic, similar texts are Reviews I do not know if they are as good as your work out.

Alphabet Nouvelle Image Google

I have a similar interest this is my page Alphabet musique read everything carefully and let me know what you think.

Alphabet Nouvelle Image Google

Extremely intriguing info, really worth promoting. Nonetheless, I recommend this kind of. Alphabet musique

Internet Recharge Tricks

You ought to basically fantastic not to mention solid advice, which means notice:
free internet tricks

holistic

I really like your blog, all the information uplifting in petrified friend and also marketing interpersonal living. Appreciate it for delivering a place intended for expressing cost-free info prefer that.. https://www.yumpu.com/en/document/view/54527650/holistic

Orange Restoration San Diego, CA

This short article is truly wonderful. That appreciates myself everything that much. This material are generally a great deal beneficial. Many thanks article author intended for giving myself this kind of decent write-up. Orange Restoration San Diego Business Information

Great tips and very easy to

Great tips and very easy to understand. This will definitely be very useful for me when I get a chance to start my blog.referencement progressif

Flight Hub Reviews

In this case you will begin it is important, it again produces a web site a strong significant internet site:
Flighthub reviews

Flight Hub Reviews

So it is interesting and very good written and see what they think about other people.
FlightHub Reviews

Sanata Tesi Frumusete Cosmetica

So luck to come across your excellent blog. Your blog brings me a great deal of fun.. Good luck with the site.
cosmetice

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.