Home / server / Questions / 1146002
Accepted
Daniel Sandberg
Asked: 2023-10-17 08:06:49 +0800 CST

Tomcat 9 - does gracefulStopAwaitMillis *properly* wait for requests to finish when using non-blocking IO connector?

  • 772

How does Tomcat 9.0.80+ handle graceful shutdown when using the NIO connector?

<Connector port="8080" protocol="HTTP/1.1"
       connectionTimeout="20000" 
       gracefulStopAwaitMillis="15000" 
       bindOnInit="false" 
       protocol="org.apache.coyote.http11.Http11NioProtocol" />

I'm already using HAproxy with retries, redispatch and health checks. I have also configured Tomcat to use graceful shutdown and late binding (as per recommendation) to let any reverse proxy/load balancer redispatch to another backend. But I'm unsure if it actually waits for in-process requests to finish processing before exiting. My testing indicates so, but I occasionally get a HTTP 502 from the public facing ALB (TLS, WAF) that sits in-front of HAproxy(advanced routing, API gateway) and then a whole bunch of Tomcat servers that get rolling restarts.

tomcat

1 Answers

  1. Best Answer

    Conclusion

    It does not guarantee that all requests will be processed!

    As far as I can tell this is what happens:

    1. A request arrives at the reverse proxy and is forwarded to a backend server.
    2. Th request is placed in a queue and picked up by a thread for processing.
    3. Another request is forwarded to Tomcat and is placed in Tomcat's request queue.
    4. Before it has time to be picked up by a processing thread, a shutdown is initiated.
    5. The original request that is already being processed by a thread will finish processing (within the gracefulStopMillis timeout).
    6. The second request is still stuck in the queue and will be dropped.
    7. The reverse proxy/loadbalancer will encounter a response timeout and return a 502 to the client.
    8. There is no safe retry the reverse proxy can do unless the application is truly idempotent.
    • 0