Home / server / Questions / 580476
In Process
Nic Cottrell
Asked: 2014-03-08 04:03:54 +0800 CST

How to diagnose Tomcat 6 suddenly stop responding?

  • 772

I run a number of tomcat instances and occasionally some just stop reponding to requests - timeout on every connection.

I'm using AJP with mod_proxy in Apache 2.2.

I get a timeout via Apache/AJP through Tomcat's AJP connector, but also via the direct HTTP connector on 8080.

I have /server-status configured within Apache and it shows 16 requests currently being processed with W, 4 idle requests and 200+ open slots with no connection. My AJP connector is configured as:

<Connector port="8009" address="localhost"
           maxThreads="250" minSpareThreads="5" maxSpareThreads="15"
           connectionTimeout="1000"
           packetSize="16384"
           maxHttpHeaderSize="16384"
           enableLookups="false" redirectPort="8443"
           emptySessionPath="true" URIEncoding="UTF-8" protocol="AJP/1.3"/>

so it should have plenty of threads to accept new connections.

Using top I see CPU and wait both under 1% and the java process has 80% memory. There is 60M free mem and 200M free swap.

I set up a special threads.jsp page using

SystemThreadList stl = new SystemThreadList();
Thread[] allThreads = stl.getAllThreads();

which gives useful information, but in this state - it doesn't load either.

In catalina.log I see:

Mar 07, 2014 11:53:09 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2

and occassional activity from other web requests, but not mine.

Is there a way from the command line, or using a profiler to get a list of threads and stack traces to find out what is getting stuck?

tomcat

2 Answers

  1. Set up Tomcat to enable RMI with these options:

    -Dcom.sun.management.jmxremote.port=12346 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Djava.rmi.server.hostname=myhost

    (This is an example, not a statement of best practice for security...)

    Use jconsole which is in the JDK and connect to your server.

    • 2

  2. If Tomcat is also unresponsive through the direct HTTP connector, then the issue is not with AJP but something is amiss inside Tomcat.

    You can get a full thread dump to be written to catalina.out by sending a 

    kill -quit <tomcat PID>
    • 1