Our configuration consists of the following components (all of them are running on a single Windows Server 2003 machine):
We are load testing the server with relatively simple requests.
It seems that under load (~8000 requests we are sending per minute from our load simulator) - we have a delay between apache web server and the mod_jk component.
Make sure you configure both Apache httpd and mod_jk to handle the traffic.
This means, the number of threads(workers) that httpd has will impact your system.
Also, in high concurrency, how you configure KeepAlive is important. There is a chance that Apache is using its threads waiting for the next request on idle connections, while active connections are not being handled.
Posting your configuration files may be a good idea.
I'm coping with a problem for more than two years now. We even hired a specialist who were unable to pinpoint what the problem was.
After a random period of time (may be 2 hours or 2 weeks), Tomcat simply stops responding. The memory utilization seems pretty normal, CPU is high (100%). We get full of "Socket write error" in the logfiles but this seems to be due to the fact that endusers keep hitting reload or stop button on their browser because of the slow /no response of the server.
We use mod_jk within Apache to connect to Tomcat. When this happens, we get a "Service temporarily unavailable" on the webpage which is fired by Apache when it can't connect to Tomcat.
The only way we found out to make this work again is to reboot the whole server! Even when restarting Tomcat or Apache HTTP server, Tomcat keep using 100% CPU and no requests are serviced. We noticed, when this happens, that when opening a webpage directly on the failing server (a webpage of www.google.com for instance), the page doesn't show up correctly, some elements (images) are missing. When hitting "reload", the images are loaded, if hitting "reload" again "images" are missing again and so on.
I performed a Thread Dump when the server was in this kind of situation and didn't find anything of interest. The only thing that differs from a Thread Dump in a regular behaviour is that all TP-Processor threads are in "Object.wait()", locking a ThreadPool$ControlRunnable and waiting a ThreadPool$ControlRunnable. Is that a deadlock? I didn't find anything useful about ThreadPool$ControlRunnable.
The good news is that problems like this are not that hard to figure out what is going on.
The first symptom you see is the 100% CPU usage. Your first step would be to narrow that down.
Threads in Object.wait() are not spinning the CPU. So it is not that. However, you don't have to guess what thread is using the CPU. There are tools out there that can tell you. On Linux, for example, you can simply look this up in "top" (as just one example)
That's where I would look, but before you spend another year trying to guess, I would upgrade your Tomcat server. 6.0.10 is extremely old, it is also early in the 6.0.x series of Tomcat, so yes, it will have several bugs in it, and you could be hitting one of them.
After that, track down the thread that is causing the jump in CPU and you can move on from there.
Over the years there have been a number of connectors developed to enable Apache httpd to communicate with Tomcat that have used a variety of protocols. When searching the web for information on how to do this, it isn't unusual to stumble across some really bad, out of date advice. So first of all the only options you should consider for this are:
All of the other other options have not been supported for a number of years so you should avoid mod_jk2, mod_jserv, mod_webapp and any other module that isn't discussed here.
My experience with providing support to SpringSource customers is that a typical customer is more likely to hit a bug in mod_proxy_ajp than they are in mod_jk or mod_proxy_http. It isn't that mod_proxy_ajp is particularly buggy, I used it myself for 18 months on a production system without a single issue, but that it has a few more bugs than the other two modules. The situation is improving but at the time of writing I would rank mod_jk and mod_proxy_http above mod_proxy_ajp.
Understanding mod_jk, mod_proxy, and mod_proxy_ajp
There are three main modules available for connecting Apache to Tomcat:
This module has the load balancing capability that was not supported in mod_proxy prior to Apache 2.2. The use of mod_proxy_ajp is preferred over mod_jk for Apache 2.2 or later.
For further documentation on mod_jk, see the official documentation as provided by the Apache Software Foundation:
Working with mod_jk: http://tomcat.apache.org/tomcat-3.3-doc/mod_jk-howto.html