TomcatExpert

Hanging by a Thread—3 Steps to Troubleshooting Tomcat

posted by mthomas on March 28, 2013 06:33 AM

It is late on a Friday afternoon, and your web application has stopped responding to requests. The server is still reachable, and the Apache Tomcat process is still running–there are no errors in the logs. You want to go home but you can’t until it is fixed. What do you do?

If your answer is “restart Tomcat and hope it stays up until Monday,” then this article is for you.

Rather than keeping your fingers crossed and hoping you don’t get an angry call from your boss over the weekend, this article will provide you with some simple steps you can take to diagnose the problem. 

Step 1: What is Tomcat Doing? Thread Dumps Begin to Answer the Question

If the Tomcat process is running, then it must be doing something. The question is what is it doing when it should be responding to requests? The way to answer that question is with a thread dump–actually, a series of thread dumps. You need to take three thread dumps roughly 10 seconds apart and then compare them. I always compare them with a diff tool rather than by eye—a it is far too easy to miss subtle but important differences between the dumps.

How you generate a thread dump depends on your operating system and how you are running Tomcat. On Linux, FreeBSD, Solaris etc. use kill -3 to trigger a thread dump. On Windows use CTRL-BREAK if Tomcat is running in a console window. If Tomcat is running as a service, then the service wrapper should provide a way to trigger a thread dump. Commons Daemon (the service wrapper that ships with Tomcat) provides an option to trigger a thread dump via the system tray icon.

Thread dumps are written to standard out. One of the many reasons for not writing standard out to application logs is because the thread dumps can be much harder to extract. For the same reason, it is a really bad idea to redirect standard out to /dev/null. Extract your thread dumps into separate files, and you are ready to start examining them.

Step 2: Examining the Thread Dumps

First of all, look at dump1 and search for the word deadlock. If you have a deadlock, the thread dump will tell you. It will also point out which threads are involved in the deadlock. Deadlocks are caused when multiple threads try and obtain the same set of locks in different orders. Deadlocks are usually an application issue, sometimes a library issue and rarely a Tomcat bug. If you find a deadlock then the only option to resolve it is to restart Tomcat. There is usually nothing you can do to prevent a deadlock from reoccurring. A fix will require code changes.

If you don’t have any deadlocks, then the next step is to compare the first and second thread dumps. The thread dump shows all the threads within the JVM including those created by the JVM. The threads you are most likely to be interested in are those related to handling HTTP or AJP requests and will be named ---exec-. will be “ajp” for AJP connectors and “http” for HTTP connectors. will be one of “bio”, “nio” or “apr” and the port will be the port the connector is listening on. is a unique ID allocated to each thread that starts at zero an increments for each new thread created for the connector. You may see a large number of threads that look similar to this:

"http-nio-8080-exec-9" daemon prio=6 tid=0x00000000151fc800 nid=0xc68 waiting on condition [0x00000000199af000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00000007ad6e1ea0>

      (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await

      ( AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)

at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

This is an idle thread waiting for a request to process. It is normal to see a number of these in the stack trace. The one case where they can indicate a problem is if you see the same number of AJP threads as you have maxThreads configured for your AJP connector. That may be an indication of thread starvation as AJP uses persistent connections. You typically need to have at least as many threads available to your AJP connector as you have maximum concurrent requests configured in your reverse proxy.

Step 3: Identifying Root Cause

Normally, you need to focus on threads that include application calls. What you are looking for is a thread that is processing a request, and it hasn’t changed from one thread dump to the next. Once you have found such a thread, you’ll need to look at the source code for your application to figure out what might be causing that thread to hang. There was a nice example on the Tomcat users mailing list recently. In this thread dump, there is only one thread in application code - http-bio-8080-exec-10. That thread is in a call to org.apache.commons.pool.impl.GenericObjectPool.borrowObject which is very suggestive. It looks like the application is hanging—it is waiting to borrow a database connection from the pool, but the pool is empty. This scenario suggests root cause—a connection leak somewhere in the application. If you enable logging for abandoned connections, you should identify the source of the leak fairly quickly.

Of course, issues vary from application to application, but the general troubleshooting process is the same. When an application hangs, take three thread dumps roughly 10 seconds apart. Then, check for deadlocks following by application threads that haven’t moved on from one thread dump to the next. If you need help interpreting the thread dump, the Tomcat user mailing list is always willing to help.

Learn more:

Mark Thomas is a Senior Software Engineer for the SpringSource Division of VMware, Inc. (NYSE: VMW). Mark has been using and developing Tomcat for over six years. He first got involved in the development of Tomcat when he needed better control over the SSL configuration than was available at the time. After fixing that first bug, he started working his way through the remaining Tomcat bugs and is still going. Along the way Mark has become a Tomcat committer and PMC member, volunteered to be the Tomcat 4 & 7 release manager, created the Tomcat security pages, become a member of the ASF and joined the Apache Security Committee. He also helps maintain the ASF's Bugzilla instances. Mark has a MEng in Electronic and Electrical Engineering from the University of Birmingham, United Kingdom.

Comments

error performing thread dump

Loved the article. We have fought this problem for a while now. It happened again yesterday and I was going to use this approach to see if we could finally figure out the culprit. We have tomcat 7 and are running it as a windows service. When I try to use the tray icon to get a Thread Dump I receive the following popup error message:

Application System Error:
The system cannot find the file specified.
Unable to open the Event Mutex.

When I google this error all I find are answers relating to Remote Desktop. I get this error even if I am at the physical machine. Any idea's what is causing this or how we can get a thread dump?

It is somewhat fantastic, and

It is somewhat fantastic, and yet check out the advice at this treat. save the marriage system Review

Thanks too

I will put my Kadek Game download link using this comment, Kadek Game on Review is an endless running game, endless run hack & slash pixelated mobile game developed by Bandung Studios - Hello, Bandung Studios was reviewed by AITINESIA which is downloadable on the Google Play Store. Kadek Game

It is somewhat fantastic, and

It is somewhat fantastic, and yet check out the advice at this treat. How To Get Your Ex Back

Nice Article

It is good, a well structured article, seems like it is your original though. Keep writing and share your idea
Cara memutihkan kulit wajah secara alami dan cepat, Hijab tutorial paris 2013 simple, Aneka resep dan cara membuat kue bolu,
harga dan spesifikasi apple ipod touch 5 terbaru, cara membuat slime mudah dan praktis

Your blogs further more each

Your blogs further more each else volume is so entertaining further serviceable It appoints me befall retreat encore. I will instantly grab your rss feed to stay informed of any updates. Survive In Bed Review

On that website page, you'll

On that website page, you'll see your description, why not read through this. ra arthritis

Thanks for writing such a

Thanks for writing such a good article, I stumbled onto your blog and read a few post. I like your style of writing... How To Give A Hand Job

It is somewhat fantastic, and

It is somewhat fantastic, and yet check out the advice at this treat. How to Turn a Guy On

Great info! I recently came

Great info! I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have. The Bonding Code

This is important, though

This is important, though it's necessary to help you head over to it weblink: elementos seguridad

wertw

Watch Bad Moms Online
Watch Rings Online
Watch Ghostbusters Online
Watch Mike and Dave Need Wedding Dates Online
Watch Miracles from Heaven Online
Watch The Purge 3 Election Year Online
Watch Ratchet and Clank Online
Watch The Secret Life of Pets Online
Watch The Nice Guys Online
Watch The Jungle Book Online
Watch Alice Through the Looking Glass Online
Watch Warcraft Online
Watch The Brothers Grimsby Online
Watch The Jungle Book Online
Watch Teenage Mutant Ninja Turtles 2 Out of the Shadows Online
Watch Free State of Jones Online
Watch The Other Side of the Door Online
Watch Me Before You Online
Watch Central Intelligence Online
Watch The Secret Life of Pets Online
Watch The Nice Guys Online
Watch Captain America Civil War Online
Watch Star Trek Beyond Online
Watch Demolition Online
Watch Free State of Jones Online
Watch Now You See Me 2 Online
Watch Criminal Online
Watch Independence Day 2 Resurgence Online
Watch Star Trek Beyond Online
Watch Amityville The Reawakening Online
Watch The BFG Online
Watch Finding Dory Online
Watch Ghostbusters Online
Watch The Brothers Grimsby Online
Watch Finding Dory Online
Watch Central Intelligence Online
Watch Bad Moms Online
Watch Now You See Me 2 Online
Watch Neighbors 2 Sorority Rising Online
Watch The Huntsman Winters War Online
Watch The Conjuring 2 Online
Watch Ice Age 5 Collision Course Online
Watch Before I Wake Online
Watch The Divergent Series Allegiant Online
Watch Captain America Civil War Online
Watch Teenage Mutant Ninja Turtles 2 Out of the Shadows Online
Watch The Divergent Series Allegiant Online
Watch Jason Bourne 5 Online
Watch Midnight Special Online
Watch The Angry Birds Movie Online
Watch The Legend of Tarzan Online
Watch Jason Bourne 5 Online
Watch Money Monster Online
Watch X-Men 9 Apocalypse Online
Watch The Conjuring 2 Online
Watch The Purge 3 Election Year Online
Watch The Huntsman Winters War Online
Watch Batman v Superman Dawn of Justice Online
Watch The Angry Birds Movie Online
Watch Rings Online
Watch Batman v Superman Dawn of Justice Online
Watch Amityville The Reawakening Online
Watch The Bronze Online
Watch Criminal Online
Watch Midnight Special Online
Watch Neighbors 2 Sorority Rising Online
Watch Warcraft Online
Watch The Boss Online
Watch Independence Day 2 Resurgence Online
Watch Suicide Squad Online
Watch Demolition Online
Watch Before I Wake Online
Watch Money Monster Online
Watch X-Men 9 Apocalypse Online
Watch Mike and Dave Need Wedding Dates Online
Watch The Bronze Online
Watch Miracles from Heaven Online
Watch The Legend of Tarzan Online
Watch Ice Age 5 Collision Course Online
Watch The Other Side of the Door Online
Watch The Boss Online
Watch Suicide Squad Online
Watch The BFG Online
Watch Alice Through the Looking Glass Online 213 1231
Watch Ratchet and Clank Online
Watch Me Before You Online

sl786982

The schedule keeps changing, and with the rain, we believe it’s been put off even more. So we don’t want to promise you anything... michael spencer gilroy

sl786982

If you don’t have any

If you don’t have any deadlocks, then you need to to evaluate and contrast the first and second line places more tips here. The line dispose of reveals all the discussions within the JVM such as those designed by the JVM.

It is delayed on a Saturday

It is delayed on a Saturday mid-day, and your web program has ceased giving answers to demands agree with. The server is still obtainable, and the Apache Tomcat procedure is still running–there are no mistakes in the records. You want to go back house but you can’t until it is set. What do you do?

Getting an in-house SEO

Getting an in-house SEO advisor from a specific SEO organization is the best and cost-effective option read this post here.

Normally, you need to focus

Normally, you need to focus on threads that include application calls. What you are looking for is a thread that is processing a request, and it hasn’t changed from one thread dump to the next. Once you have found such a thread, you’ll need to look at the source code for your application to figure out what might be causing that thread to hang.

eye infections

Tomcat Freeze

Tomcat is hung with out any thread hang, all http* threads are waiting to process requests. No out of memory. Browsers / http clients are hanging for response from tomcat server.

Any help to debug this issue?

News Greek

This is very useful, although it will be important to help simply click that web page link:
cyprus

The Salvation Diet Review

This is very interesting, but it is necessary to click on this link:
Salvation Diet Review

Excellent .. Amazing .. I’ll

Excellent .. Amazing .. I’ll bookmark your blog and take the feeds also…I’m happy to find so many useful info here in the post, we need work out more techniques in this regard, thanks for sharing.
AC repair

reply

The tips and tricks that are shared here on the page to troubleshoot Tomcat proved to be a really useful one. The tips helped me to know more about the various troubleshooting steps of Tomcat. Keep up the good work guys search here

winstrol supplements

I 'd confirm that a large portion of us guests are truly enhanced to exist in a dazing spot with a huge amount of extraordinary people with to an incredible degree satisfying things. winstrol supplements

This is really interesting,

This is really interesting, You are a very skilled blogger. I’ve joined your feed and look forward to seeking more of your excellent post. Also, I’ve shared your web site in my social networks! Samenspender gesucht

Working Clash of Kings Cheats

Thank you for this formula because i am newbie and this thing really helped me a lot with creating mine clash of kings cheats

Get latest working racing rivals mod apk from JnS company website

marijuana doctors

Medical marijuana card doctors providing 420 evaluations, medical marijuana recommendations and renewals for medical marijuana patients in Hollywood Los Angeles California. Our medical marijuana doctors are the best in Los Angeles California. Our medical marijuana doctors specialize in out of state patients, renewals, and growing marijuana. If you are wondering how to get a medical marijuana card with or without medical records give our medical marijuana Doctor a call and learn about the pros and cons of a medical card. Under prop 215 growers permits are available that exempt a patient with a large medical marijuana requirement from SB 420. marijuana doctors

I was very pleased to find

I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post. BaByliss Pro BABP6685N Hair Dryer Review

GAME HACK

pou hack I wrote about a similar issue, I give you the link to my site. clash of clans astuce

muneer

Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have. Keep up the good work you are doing here.website design

guest post service

Hello there, You’ve performed an excellent job. I’ll certainly dig it and for my part recommend to my friends. I’m confident they’ll be benefited from this website.guest post service

alex

I want plenty of articles and blogs please upload shortly.Bound Gravel

workout room

Walking in the presence of giants here. Cool thinking all around! workout room

BlackMen

I always compare them with a diff tool rather than by eye—a it is far too easy to miss subtle but important differences between the dumps.cccam server

Muneer

Click here to start blogging instantly!!! Good day very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your site and take the feeds additionally?I am glad to find numerous useful info right here within the post, we want work out extra techniques in this regard, thank you for sharing

I'm impressed, I must say.

I'm impressed, I must say. Very rarely do I come across a blog thats both informative and entertaining, and let me tell you, you ve hit the nail on the head. Your blog is important..best domain parking company

gta online game

I'm impressed, I must say. Very rarely do I come across a blog thats both informative and entertaining, and let me tell you, you ve hit the nail on the head.Game Network was a European online game and television network company. It was initially owned by Digital Bros. group, but was later sold to Cellcast Group. Game Network was, during its broadcast, the only television channel in Europe dedicated to covering multimedia entertainment, and generally focused on video games.

Play GTA Vice City Online
Play GTA San Andreas Online
Jugar Vice City
Jugar San Andreas
pes 2007 download
pes 2010 pc
Telecharger Minecraft Gratuit
Telecharger Subway Surfers Gratuit
Telecharger FIFA 12 PC
Telecharger IDM Gratuit

dowry

I just thought it may be an idea to post incase anyone else was having problems researching but I am a little unsure if I am allowed to put names and addresses on here. dowry act 498a

pheromones that work

I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details. pheromones that work

During this website, you will

During this website, you will see this shape, i highly recommend you learn this review. computer classes in houston

You ought to basically

You ought to basically fantastic not to mention solid advice, which means notice: houston bakery

BlackMen

Extract your thread dumps into separate files, and you are ready to start examining them.iptv server

OC Housing News

On my website you'll see similar texts, write what you think.
Real Estate Market News

Jual Laptop Notebook PC HP Lenovo Printer Tinta Toner Epson Mura

Make the most of mainly premium substances - you will find him or her for:
Jual PC HP

Watch Latest Pakistani Talk Shows and Dramas Online HD at ViiTV

This is very appealing, however , it is very important that will mouse click on the connection:
Watch Latest pakistani talk shows

ali

Excellent post.I want to thank you for this informative read, I really appreciate sharing this great post. Keep up your work wimpernverlängerung

TRUTH About PhenQ: Reviews, Ingredients, Side Effects

This is very interesting, but it is necessary to click on this link:
best diet pills 2016

Global Best Beauty Trading Ltd

Your blogs further more each else volume is so entertaining further serviceable It appoints me befall retreat encore. I will instantly grab your rss feed to stay informed of any updates.
Seamless Tape Extensions

alex

I honer your kindness in sharing your incredible knowledge with us.West Palm Beach SEO

jhonwattson

هذا هو المفيد للغاية وقطعة العملي من العمل على الاستثمار. لقد وجدت أنها مثمرة للغاية. fxprotrade, افكس بروتريد

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.