Home / server / Questions / 9110
Accepted
Nik Reiman
Asked: 2009-05-18 08:11:33 +0800 CST

Can a naughty database app crash tomcat?

  • 772

We've got a number of web services running through Tomcat which use hibernate/mysql. I suspect that some of them have incorrectly configured connection pools, as after several hours some of the individual apps run out of connections and stop responding. We have been making changes to the connection pool service (in this case, C3P0), but we still need to leave the old versions of the applications on the server for backwards compatibility.

At any rate, I suspect that these applications are also hurting tomcat's overall stability. About once a week, our server stops responding entirely and is unable to serve even static pages. After restarting the service, everything works again for another few days or so. Looking through the logs reveals hardly any uncaught exceptions, so I'm not sure what would be causing tomcat to crash. Nothing remarkable is listed in the error logs before the server quits responding, sadly.

We are also considering switching to JBoss, as it is a bit more "enterprisey", but I'm not convinced it will solve these problems. Is there any compelling reason to switch web platforms, or should I debug further within our own webapps? Also, is it possible for the webapp to crash the application server by doing something bad?

Server config: Windows 2003 Server, Tomcat 6.0.18 + blazeDS 3.0, Hibernate 3.2.

tomcat web-applications hibernate

4 Answers

  1. Best Answer

    I don't think anybody will have the answer to your problem, but only leads & ideas. Here are some:

    • you need robots that will check the health of each part of your service. (testing a single connection to your database, getting a static web page, getting a dynamic web page...). This way, you'll see what breaks first or response time increase.

    • do you have a monitoring/stats service? You need to keep track of the "number of active database connections", "number of active web sessions", "number of tomcat threads", "memory available", CPU...

    My advice, there is no tomcat process left because they're all stuck waiting for a resource (maybe a database connection, or they're just an infinite loop!). The tools I listed before will definitely help you understand why your server is dying slowly every week.

    1. run netstat on your server and look at the number of connections to the database server (and check it against your pool size and database server capacity).
    2. run jstack on your app server and cut/grep/sort them to see what your threads are doing.
    • 2

  2. If serving the static pages does not require any database access, then it seems unlikely that it is a database resource issue as such. It may be that all of the pooled threads are stuck somewhere, such as waiting for the database drive or in a deadlock. The first thing I'd do is get a snapshot of the stack traces with jstack. You can further look at the process with visualvm or jconsole.

    • 1

  3. Just wanted to add that it's quite common for table locking issues with MyISAM tables can easily cause DB connections to pile up and cause the application waiting for those results to sit and sit.

    You may want to check out the MySQL process list to see if there are a lot of queries sitting in a locked state.

    # mysqladmin processlist

    -- or --

    mysql> show processlist;

    If locking is the problem, you will want to see if changing the storage engine on the problem tables from MyISAM to InnoDB is feasible.

    • 1

  4. If you install the lambda probe webapp (get the 1.7 beta) you can get thread-level monitoring; keeping an eye on this will tell you when threads are stuck waiting for the database, as well as a host of other useful diagnostics.

    It's a little old but still works fine in recent tomcat releases.

    • 1