TomcatExpert

Apache Tomcat hang after a random period of time

Hello,

I'm coping with a problem for more than two years now. We even hired a specialist who were unable to pinpoint what the problem was.

After a random period of time (may be 2 hours or 2 weeks), Tomcat simply stops responding. The memory utilization seems pretty normal, CPU is high (100%). We get full of "Socket write error" in the logfiles but this seems to be due to the fact that endusers keep hitting reload or stop button on their browser because of the slow /no response of the server.

We use mod_jk within Apache to connect to Tomcat. When this happens, we get a "Service temporarily unavailable" on the webpage which is fired by Apache when it can't connect to Tomcat.

The only way we found out to make this work again is to reboot the whole server! Even when restarting Tomcat or Apache HTTP server, Tomcat keep using 100% CPU and no requests are serviced. We noticed, when this happens, that when opening a webpage directly on the failing server (a webpage of www.google.com for instance), the page doesn't show up correctly, some elements (images) are missing. When hitting "reload", the images are loaded, if hitting "reload" again "images" are missing again and so on.

I performed a Thread Dump when the server was in this kind of situation and didn't find anything of interest. The only thing that differs from a Thread Dump in a regular behaviour is that all TP-Processor threads are in "Object.wait()", locking a ThreadPool$ControlRunnable and waiting a ThreadPool$ControlRunnable. Is that a deadlock? I didn't find anything useful about ThreadPool$ControlRunnable.

We have splitted our web applications on other servers, used other configurations (mod_proxy), virtualized the servers, unvirtualized the servers, used Linux, used Windows... I am lost..

Here is our actual configuration:

  • Apache 2.2.3
  • mon_jk
  • Tomcat 6.0.10 as a Windows Service
  • Windows Server 2003 SP2
  • Connects to an IBM DB2 Database on iSeries

asked by eric

question

The good news is that problems like this are not that hard to figure out what is going on.

The first symptom you see is the 100% CPU usage. Your first step would be to narrow that down.

Steps
  1. What process is using the CPU
  2. What thread within that process is using the CPU
  3. What is that thread doing at the time the CPU is being consumed

Threads in Object.wait() are not spinning the CPU. So it is not that. However, you don't have to guess what thread is using the CPU. There are tools out there that can tell you. On Linux, for example, you can simply look this up in "top" (as just one example)

That's where I would look, but before you spend another year trying to guess, I would upgrade your Tomcat server. 6.0.10 is extremely old, it is also early in the 6.0.x series of Tomcat, so yes, it will have several bugs in it, and you could be hitting one of them.

After that, track down the thread that is causing the jump in CPU and you can move on from there.

answer

answered by fhanik on June 16, 2011 11:33 AM

Filip Hanik is a Senior Software Engineer for the SpringSource Division of VMware, Inc. (NYSE: VMW) and a key participant in the company's Apache Tomcat initiatives. Filip brings 15 years of extensive experience in architecture, design and development of distributed application frameworks and containers and is recognized for his top-quality system development skills and continuous participation of Open Source development projects. Filip is an Apache Software Foundation member and a committer to the Apache Tomcat project where he is a leading authority on Tomcat clustering and a key contributor to the core of the platform. Filip has made contributions to software initiatives for Walmart.com, Sony Music, France Telecom and has held a variety of senior software engineering positions with technology companies in both the United States and Sweden. He received his education at Chalmers University of Technology in Gothenburg, Sweden where he majored in Computer Science and Computer Engineering.

Comments

Sujeet

I have also faced issues with the Tomcat server and I am happy that I have reached the proper place to get a fix for the problem. I hope that after doing the steps that are explained here on the page, I will be able to get rid of the problem. directv family package

SEO usually is expensive, and

SEO usually is expensive, and as such it is unrealistic to think you can do everything possible in every facet of internet promotion. So often you need to make choices, and some may be tempted to concentrate on link-building instead of material Online Gift Registry.

I am extremely happy to know

I am extremely happy to know that you guys have finally found the solution to the particular problem. I just say that superior paper presents many regarding the Apache tomcat.

Same Problem

I was also having the same problem, seems that it is something to take with RAM as it takes lots of memory. pune pest control

great

If you do follow these steps carefully, then you would not have anything to worry about. I advise that you do seek some professional help from here or elsewhere if you are in doubt. There is a high chance that you might mess things up. anti snore mouthpiece

gta online

Game Network was a European online game and television network company. It was initially owned by Digital Bros. group, but was later sold to Cellcast Group. Game Network was, during its broadcast, the only television channel in Europe dedicated to covering multimedia entertainment, and generally focused on video games.

Play GTA Vice City Online
Play GTA San Andreas Online
Jugar Vice City
Jugar San Andreas
pes 2007 download
pes 2010 pc
Telecharger Minecraft Gratuit
Telecharger Subway Surfers Gratuit
Telecharger FIFA 12 PC
Telecharger IDM Gratuit

Hello. I think rebooting the

Hello. I think rebooting the whole server may help in your case.
At least this is the only solution that came to my mind.
_____________________________
writemyessayforcheap.blogspot.com - team that provides assistance and takes care of you.

GAME HACK

castle clash strategy For many people this is important, so check out my profile: clash of clans gemmes.com

Any solution

I just started using Tomcat (7) and it seems to be doing this same thing. Did you ever find a solution or a cause to the problem?

Tomcat 5.0 Hangs

I have problem with tomcat 5.0 on Windows XP hangs after 10-15 days. It gives follwoing error:
2012-03-05 12:04:51 StandardWrapperValve[jsp]: Servlet.service() for servlet jsp threw exception
java.lang.IllegalStateException: getOutputStream() has already been called for this response
at org.apache.coyote.tomcat5.CoyoteResponse.getWriter(CoyoteResponse.java:614)
at org.apache.coyote.tomcat5.CoyoteResponseFacade.getWriter(CoyoteResponseFacade.java:210)
at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:167)
at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:160)
at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:235)
at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:160)
at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:120)
at org.apache.jsp.jsp.TotalConsumptionReport_jsp._jspService(TotalConsumptionReport_jsp.java:344)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:245)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:199)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:164)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:149)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:156)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:972)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:206)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:833)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:732)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:619)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:688)
at java.lang.Thread.run(Thread.java:536)

Post new comment

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.