Home / user-472919

Daniel Sandberg's questions

Martin Hope
Daniel Sandberg
Asked: 2023-10-17 08:06:49 +0800 CST
  • 5

How does Tomcat 9.0.80+ handle graceful shutdown when using the NIO connector?

<Connector port="8080" protocol="HTTP/1.1"
       connectionTimeout="20000" 
       gracefulStopAwaitMillis="15000" 
       bindOnInit="false" 
       protocol="org.apache.coyote.http11.Http11NioProtocol" />

I'm already using HAproxy with retries, redispatch and health checks. I have also configured Tomcat to use graceful shutdown and late binding (as per recommendation) to let any reverse proxy/load balancer redispatch to another backend. But I'm unsure if it actually waits for in-process requests to finish processing before exiting. My testing indicates so, but I occasionally get a HTTP 502 from the public facing ALB (TLS, WAF) that sits in-front of HAproxy(advanced routing, API gateway) and then a whole bunch of Tomcat servers that get rolling restarts.

tomcat
Martin Hope
Daniel Sandberg
Asked: 2021-05-18 11:03:50 +0800 CST
  • 1

Anyone been hit by the NPE bug when using JNDIRealm/ActiveDirectory in Tomcat 9.0.46 and have a workaround?

Context

I had to upgrade from 9.0.43 due to a concurrency bug where the socket input stream gets corrupted (and mixing up request bodies) when keep-alive/re-use is enabled. Log was filled with "Error parsing HTTP request header" and "HTTP method names must be tokens 000x000x000x000x000x...", etc. After upgrade to 9.0.46 this issue is resolved but now our ActiveDirectory login is broken instead.

Log

17-May-2021 14:31:49.405 INFO [Catalina-tomcat-workers-thread7] org.apache.catalina.realm.JNDIRealm.authenticate Exception performing authentication. Retrying...
    java.lang.NullPointerException
            at org.apache.catalina.realm.JNDIRealm.doAttributeValueEscaping(JNDIRealm.java:2884)
            at org.apache.catalina.realm.JNDIRealm.getRoles(JNDIRealm.java:1892)
            at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1350)
            at org.apache.catalina.realm.JNDIRealm.authenticate(JNDIRealm.java:1232)
            at org.apache.catalina.realm.CombinedRealm.authenticate(CombinedRealm.java:191)
            at org.apache.catalina.realm.CombinedRealm.authenticate(CombinedRealm.java:191)
            at org.apache.catalina.realm.LockOutRealm.authenticate(LockOutRealm.java:154)
            at org.apache.catalina.authenticator.BasicAuthenticator.doAuthenticate(BasicAuthenticator.java:101)
            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:632)

Bug

According to the bug report (https://bz.apache.org/bugzilla/show_bug.cgi?id=65308) there is an issue with userRoleAttribute being empty/null. Guess that won't get released for another 3-4 weeks.

The Tomcat documentation (https://tomcat.apache.org/tomcat-9.0-doc/realm-howto.html#JNDIRealm) says that:

roleSearch - the LDAP search filter for selecting role entries.
It optionally includes pattern replacements "{0}" for the distinguished name and/or "{1}" for the username and/or "{2}" for an attribute from user's directory entry, of the authenticated user.
Use userRoleAttribute to specify the name of the attribute that provides the value for "{2}".

I'm not using "{2}" in the roleSearch attribute so I don't have a need to set userRoleAttribute. I have nevertheless tried to set it to something non-empty, but to no avail.

I need a workaround until such time they release 9.0.47. Any takers?

tomcat active-directory bug
Martin Hope
Daniel Sandberg
Asked: 2020-08-19 13:58:45 +0800 CST
  • 2

Background

We currently have a local deb mirror setup using Aptly snapshots due to regulatory reasons (for example: servers are not allowed to fetch packages over the internet). All of this works just fine for normal upgrades. When we need to do a release upgrade (from Ubuntu 18.04 to 20.04) we can normally just replace the VM, but some servers require us to do an in-place release upgrade using our mirror.

Current state

We have configured /etc/update-manager/meta-release on our nodes to point to our mirror but of-course there is no meta-release and meta-release-lts files in our Aptly. If we then copy the official meta-release files to be served from our Aptly it still won't work because it refers to a bunch of different external URLs, installers, documentations, etc. and even if we would change the URLs to point to our Aptly all those files would not be there. So we would need to be copy and serve those too.

Solution?

Do we need to parse the meta-release files, download all the installers, etc. serve them and generate our own meta-release files from that? There must be a simpler way of doing this! Do we have to build this from scratch or is there tooling available?

https://www.aptly.info/

PS. We are thinking of switching to Pulp 3 since Aptly seems to be partially abandoned, not that we know if that would help with the release upgrade issue.

ubuntu apt upgrade aptly
Martin Hope
Daniel Sandberg
Asked: 2018-06-07 08:35:51 +0800 CST
  • 0

What is the issue?

After upgrading instances to Ubuntu 16.04 LTS and OpenJDK 8 the GC frequency can, after hours of normal operation, suddenly increase by orders of magnitude ending up using less than 10% of the HEAP.

We have analyzed the GC logs and can find nothing strange except the frequency increase. Even gceasy.io says everything is awesome.

It does not seem to effect performance negatively but is still worrying behaviour and Nagios/OP5 is not happy. The allocation rate is only 50mb/s and the throughput and latency is nominal.

We have tried to scrap the entire cluster, install everything from scratch, multiple variations of different GC and memory settings and the issue still comes back.

Did it work before?

Yes. Other instances running the same versions of the applications on Ubuntu 14.04 LTS, OpenJDK 7u151 works as expected with normal frequency and HEAP usage.

Questions

  1. Is this just how OpenJDK 8 behaves?
  2. Is this an Ubuntu or OpenJDK issue?
  3. Is pinning the HEAP with Xms our only option?
  4. (What is actually going on here?)

Instances

VM: AWS (4GB, 2 cores)
Ubuntu: Ubuntu 16.04.4 LTS
Kernel: 4.4.0-1060-aws
Java:
- OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
- OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
Tomcat 7.0.8x running with NIO connector and 150 threads

Command line (latest iteration)

export CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m"
export CATALINA_OPTS="$CATALINA_OPTS -XX:NewRatio=1"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+UseParallelOldGC"
export CATALINA_OPTS="$CATALINA_OPTS -XX:ThreadStackSize=256k"
export CATALINA_OPTS="$CATALINA_OPTS -server"
export CATALINA_OPTS="$CATALINA_OPTS -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Xloggc:/var/log/tomcat7/gc.log"

After 48 hours performance test

$ free
              total        used        free      shared  buff/cache   available
Mem:        3801048     1248348     1959488        5280      593212     2291640
Swap:             0           0           0

This is a snapshot of the past 24 hours of performance testing: - 05:30 sudden drop in HEAP usage and increase in GC - 10:00 increase the load by x2

performance test graphs

ubuntu