I arrived at work today to find that tomcat6 won't start. When trying to stop and start with the following command(s):
service tomcat6 stop ; > /var/log/tomcat6/catalina.out && service tomcat6 start && less +F /var/log/tomcat6/catalina.out
It reports back OK
for starting tomcat, but using ps
, no tomcat process is shown. The catalina.out file doesn't show anything at all either, completely blank. Upon looking at /var/log/tomcat6/tomcat6-initd.log, I see two lines repeated over and over on every attempt:
/usr/sbin/tomcat6: line 60: ${2}: ambiguous redirect
/usr/sbin/tomcat6: line 30: ${2}: ambiguous redirect
Looking at /usr/sbin/tomcat6, both lines read:
${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \
-classpath "$CLASSPATH" \
-Dcatalina.base="$CATALINA_BASE" \
-Dcatalina.home="$CATALINA_HOME" \
-Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
-Djava.io.tmpdir="$CATALINA_TMPDIR" \
-Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
-Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
org.apache.catalina.startup.Bootstrap start \
>> ${2} 2>&1 &
The first line being the same for both 60 & 30. My $JAVA_OPTS
reads:
JAVA_OPTS="-server -Xss128K -Xms3G -Xmx3G -XX:MaxPermSize=256M -Dambiguous.url=https://ambiguous.url.com"
and so far as I can tell (uncertain where to look, honestly), $CATALINA_OPTS
is null, but I could be mistaken. One things for certain, I haven't adjusted its value, and am confident that the value has nothing to do with the ambiguous redirect issues.
UPDATE: By updating our tomcat6 package to the latest version, I now do not receive the errors seen above. In fact, the /usr/sbin/tomcat6 file reads entirely different around the areas it was pointing me to before:
${JAVACMD} $JAVA_OPTS $CATALINA_OPTS \
-classpath "$CLASSPATH" \
-Dcatalina.base="$CATALINA_BASE" \
-Dcatalina.home="$CATALINA_HOME" \
-Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" \
-Djava.io.tmpdir="$CATALINA_TMPDIR" \
-Djava.util.logging.config.file="${CATALINA_BASE}/conf/logging.properties" \
-Djava.util.logging.manager="org.apache.juli.ClassLoaderLogManager" \
org.apache.catalina.startup.Bootstrap start \
>> ${CATALINA_BASE}/logs/catalina.out 2>&1 &
if [ ! -z "$CATALINA_PID" ]; then
echo $! > $CATALINA_PID
fi
Now it gets even more complicated, as tomcat6-initd.log does not report anything at all, yet the service still doesn't start up. Running the init script with a -x
results in the following output on startup:
# service tomcat6 start
+ '[' -r /lib/lsb/init-functions ']'
+ . /lib/lsb/init-functions
++ lsb_release -i -s
+ DISTRIB_ID=Scientific
++ basename /etc/init.d/tomcat6
+ NAME=tomcat6
+ unset ISBOOT
+ '[' t = S -o t = K ']'
+ '[' -x /sbin/runuser ']'
+ SU='/sbin/runuser -s /bin/sh'
+ TOMCAT_CFG=/etc/tomcat6/tomcat6.conf
+ '[' -r /etc/tomcat6/tomcat6.conf ']'
+ . /etc/tomcat6/tomcat6.conf
++ CATALINA_BASE=/usr/share/tomcat6
++ CATALINA_HOME=/usr/share/tomcat6
++ JASPER_HOME=/usr/share/tomcat6
++ CATALINA_TMPDIR=/var/cache/tomcat6/temp
++ JAVA_OPTS='-Xms3072m -Xmx3072m -Xss512m -XX:MaxPermSize=512m -server -Dambiguous.url=https://ambiguous.url.com'
++ TOMCAT_USER=root
++ SECURITY_MANAGER=false
++ SHUTDOWN_WAIT=30
++ SHUTDOWN_VERBOSE=false
++ CATALINA_PID=/var/run/tomcat6.pid
+ '[' -r /etc/sysconfig/tomcat6 ']'
+ . /etc/sysconfig/tomcat6
+ CONNECTOR_PORT=8080
+ TOMCAT_SCRIPT=/usr/sbin/tomcat6
+ TOMCAT_PROG=tomcat6
+ TOMCAT_USER=root
+ TOMCAT_LOG=/var/log/tomcat6/tomcat6-initd.log
+ RETVAL=0
+ RETVAL=0
+ case "$1" in
+ start
+ echo -n 'Starting tomcat6: '
Starting tomcat6: + '[' 0 '!=' 0 ']'
+ '[' -f /var/lock/subsys/tomcat6 ']'
+ '[' -f /var/run/tomcat6.pid ']'
+ read kpid
+ '[' -d /proc/ ']'
+ log_success_msg
+ /etc/redhat-lsb/lsb_log_message success
[ OK ]
+ '[' Scientific = MandrivaLinux ']'
+ RETVAL=0
+ return
+ exit 0
If I were to run the exact same setup on a separate server where tomcat works (there's multiple virtual machines running as a cluster), the output reads:
# service tomcat6 start
+ '[' -r /lib/lsb/init-functions ']'
+ . /lib/lsb/init-functions
++ lsb_release -i -s
+ DISTRIB_ID=Scientific
++ basename /etc/init.d/tomcat6
+ NAME=tomcat6
+ unset ISBOOT
+ '[' t = S -o t = K ']'
+ '[' -x /sbin/runuser ']'
+ SU='/sbin/runuser -s /bin/sh'
+ TOMCAT_CFG=/etc/tomcat6/tomcat6.conf
+ '[' -r /etc/tomcat6/tomcat6.conf ']'
+ . /etc/tomcat6/tomcat6.conf
++ CATALINA_BASE=/usr/share/tomcat6
++ CATALINA_HOME=/usr/share/tomcat6
++ JASPER_HOME=/usr/share/tomcat6
++ CATALINA_TMPDIR=/var/cache/tomcat6/temp
++ JAVA_OPTS='-Xms3072m -Xmx3072m -Xss512m -XX:MaxPermSize=512m -server -Dambiguous.url=https://ambiguous.url.com'
++ TOMCAT_USER=root
++ SECURITY_MANAGER=false
++ SHUTDOWN_WAIT=30
++ SHUTDOWN_VERBOSE=false
++ CATALINA_PID=/var/run/tomcat6.pid
+ '[' -r /etc/sysconfig/tomcat6 ']'
+ . /etc/sysconfig/tomcat6
+ CONNECTOR_PORT=8080
+ TOMCAT_SCRIPT=/usr/sbin/tomcat6
+ TOMCAT_PROG=tomcat6
+ TOMCAT_USER=root
+ TOMCAT_LOG=/var/log/tomcat6/tomcat6-initd.log
+ RETVAL=0
+ RETVAL=0
+ case "$1" in
+ start
+ echo -n 'Starting tomcat6: '
Starting tomcat6:
+ '[' 0 '!=' 0 ']'
+ '[' -f /var/lock/subsys/tomcat6 ']'
+ export CATALINA_PID=/var/run/tomcat6.pid
+ CATALINA_PID=/var/run/tomcat6.pid
+ touch /var/run/tomcat6.pid
+ '[' 0 -eq 0 -a 0 -eq 0 ']'
+ chown root:root /var/run/tomcat6.pid
+ '[' 0 -eq 0 ']'
+ touch /var/log/tomcat6/tomcat6-initd.log
+ '[' 0 -eq 0 -a 0 -eq 0 ']'
+ chown root:root /var/log/tomcat6/tomcat6-initd.log
+ parseOptions
+ options=
++ awk '!/^#/ && !/^$/ { ORS=" "; print "export ", $0, ";" }' /etc/tomcat6/tomcat6.conf
+ options=' export CATALINA_BASE="/usr/share/tomcat6" ; export CATALINA_HOME="/usr/share/tomcat6" ; export JASPER_HOME="/usr/share/tomcat6" ; export CATALINA_TMPDIR="/var/cache/tomcat6/temp" ; export JAVA_OPTS="-Xms3072m -Xmx3072m -Xss512m -XX:MaxPermSize=512m -server -Dambiguous.url=https://ambiguous.url.com" ; export TOMCAT_USER="root" ; export SECURITY_MANAGER="false" ; export SHUTDOWN_WAIT="30" ; export SHUTDOWN_VERBOSE="false" ; export CATALINA_PID="/var/run/tomcat6.pid" ; '
+ '[' -r /etc/sysconfig/tomcat6 ']'
++ awk '!/^#/ && !/^$/ { ORS=" ";
print "export ", $0, ";" }' /etc/sysconfig/tomcat6
+ options=' export CATALINA_BASE="/usr/share/tomcat6" ; export CATALINA_HOME="/usr/share/tomcat6" ; export JASPER_HOME="/usr/share/tomcat6" ; export CATALINA_TMPDIR="/var/cache/tomcat6/temp" ; export JAVA_OPTS="-Xms3072m -Xmx3072m -Xss512m -XX:MaxPermSize=512m -server -Dambiguous.url=https://ambiguous.url.com" ; export TOMCAT_USER="root" ; export SECURITY_MANAGER="false" ; export SHUTDOWN_WAIT="30" ; export SHUTDOWN_VERBOSE="false" ; export CATALINA_PID="/var/run/tomcat6.pid" ; '
+ TOMCAT_SCRIPT=' export CATALINA_BASE="/usr/share/tomcat6" ; export CATALINA_HOME="/usr/share/tomcat6" ; export JASPER_HOME="/usr/share/tomcat6" ; export CATALINA_TMPDIR="/var/cache/tomcat6/temp" ; export JAVA_OPTS="-Xms3072m -Xmx3072m -Xss512m -XX:MaxPermSize=512m -server -Dambiguous.url=https://ambiguous.url.com" ; export TOMCAT_USER="root" ; export SECURITY_MANAGER="false" ; export SHUTDOWN_WAIT="30" ; export SHUTDOWN_VERBOSE="false" ; export CATALINA_PID="/var/run/tomcat6.pid" ; /usr/sbin/tomcat6'
+ '[' 0 -eq 0 -a false = true ']'
+ '[' 0 -eq 0 ']'
+ /sbin/runuser -s /bin/sh - root -c ' export CATALINA_BASE="/usr/share/tomcat6" ; export CATALINA_HOME="/usr/share/tomcat6" ; export JASPER_HOME="/usr/share/tomcat6" ; export CATALINA_TMPDIR="/var/cache/tomcat6/temp" ; export JAVA_OPTS="-Xms3072m -Xmx3072m -Xss512m -XX:MaxPermSize=512m -server -Dambiguous.url=https://ambiguous.url.com" ; export TOMCAT_USER="root" ; export SECURITY_MANAGER="false" ; export SHUTDOWN_WAIT="30" ; export SHUTDOWN_VERBOSE="false" ; export CATALINA_PID="/var/run/tomcat6.pid" ; /usr/sbin/tomcat6 start'
+ '[' 0 -eq 0 ']'
+ log_success_msg
+ /etc/redhat-lsb/lsb_log_message success
[ OK ]
+ touch /var/lock/subsys/tomcat6
+ '[' Scientific = MandrivaLinux ']'
+ exit 0
And just to be thorough, the start()
function in /etc/init.d/tomcat6
reads:
function start() {
echo -n "Starting ${TOMCAT_PROG}: "
if [ "$RETVAL" != "0" ]; then
log_failure_msg
return
fi
if [ -f "/var/lock/subsys/${NAME}" ]; then
if [ -f "/var/run/${NAME}.pid" ]; then
read kpid < /var/run/${NAME}.pid
# if checkpid $kpid 2>&1; then
if [ -d "/proc/${kpid}" ]; then
log_success_msg
if [ "$DISTRIB_ID" = "MandrivaLinux" ]; then
echo
fi
RETVAL="0"
return
fi
fi
fi
# fix permissions on the log and pid files
export CATALINA_PID="/var/run/${NAME}.pid"
touch $CATALINA_PID 2>&1 || RETVAL="4"
if [ "$RETVAL" -eq "0" -a "$?" -eq "0" ]; then
chown ${TOMCAT_USER}:${TOMCAT_USER} $CATALINA_PID
fi
[ "$RETVAL" -eq "0" ] && touch $TOMCAT_LOG 2>&1 || RETVAL="4"
if [ "$RETVAL" -eq "0" -a "$?" -eq "0" ]; then
chown ${TOMCAT_USER}:${TOMCAT_USER} $TOMCAT_LOG
fi
parseOptions
if [ "$RETVAL" -eq "0" -a "$SECURITY_MANAGER" = "true" ]; then
$SU - $TOMCAT_USER -c "${TOMCAT_SCRIPT} start-security" \
>> ${TOMCAT_LOG} 2>&1 || RETVAL="4"
else
[ "$RETVAL" -eq "0" ] && $SU - $TOMCAT_USER -c "${TOMCAT_SCRIPT} start" >> ${TOMCAT_LOG} 2>&1 || RETVAL="4"
fi
if [ "$RETVAL" -eq "0" ]; then
log_success_msg
touch /var/lock/subsys/${NAME}
else
log_failure_msg "Error code ${RETVAL}"
fi
if [ "$DISTRIB_ID" = "MandrivaLinux" ]; then
echo
fi
}
Typed
reboot
into the console, mashed the enter key with considerable force using my forehead, and the problem went away.