Home / server / Questions / 629354
In Process
fuero
Asked: 2014-09-18 15:13:26 +0800 CST

Java Spring application working on one system, dying on almost identical one

  • 772

Introduction

I'm running a third party Java Spring application on RHEL, that refuses to run on machines cloned from a certain RHEL 6.5 template. It just dies on startup with a bogus exception (see below).

The versions of the software in question:

RHEL 6.5 
Tomcat 7.0.41 (rolled our own, almost vanilla)
Spring 3.1.4.RELEASE (shipped with application)
java version "1.7.0_65"
OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
SELinux Status: Permissive

I've changed class names to protect the innocent, I highly doubt that the actual software is significant.

Copying tomcat and the instance directory from the broken box to my Linux desktop as is and just running it works.

While the setup I am describing is considered a test environment, I did experience this problem on our prod box, which I had to install cleanly from DVD to make the issue disappear. The mentioned setup is the one I re-encountered the problem during puppet module development and managed to conserve and reproduce this problem in.

Setup

DBs the application uses are kept local (h2) and are set up in tomcat's server.xml.

2 RHEL 6.5 templates, both setup for vagrant.

  • Broken template: minimal install straight from DVD. Manual setup steps taken to set up template for puppet autosigning, and vagrant template checklist steps.
  • Working template: Kickstart minimal setup from DVD, Vagrant/VirtualBox customization via Shellscripts (done with packer)

1 box from each template is set up via vagrant, both apply the exact same puppet catalog from our puppet master.

Problem

The application dies on startup in the broken template with this exception: 

[17.09.2014 19:34:00,290] ERROR                  ContextLoader ( 318) | Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'soaFormFolderService' defined in file [/var/www/apps/content/tomcat/afs-forms/webapps/formserver/WEB-INF/classes/com/aforms2web/afs/formserver/service/
SoaFormFolderService.class]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:943)
[...]
Caused by: java.lang.ExceptionInInitializerError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:87)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:990)
        ... 26 more
Caused by: java.lang.NullPointerException
        at com.corp.Application.getMessageSource(Application.java:234)
        at com.corp.FooService.<clinit>(FooService.java:23)
        ... 33 more

The code in question looks like this:

com.corp.FooService

@Service
public class FooService
{
  protected static MessageSource messageSource = Application.getInstance().getMessageSource();

com.corp.Application

public class Application
{
  private static Application instance = null;

  public static Application getInstance()
  {
    if (instance == null) {
      instance = new Application();
    }
    return instance;
  }
  [...]
  public MessageSource getMessageSource()
  {
    return ((MessageSource)this.applicationContext.getBean("messageSource", MessageSource.class));
  }

It runs without a hitch in the second box.

What could cause this to happen?

Investigation status

diff style yum list installed output:

--- pkgs.broken
+++ pkgs       
-b43-openfwwf.noarch-5.2-4.el6
-dkms.noarch-2.2.0.3-25.el6
-efibootmgr.x86_64-0.5.4-11.el6
+gcc-c++.x86_64-4.4.7-4.el6
-kernel-devel.x86_64-2.6.32-431.23.3.el6
+kernel-devel.x86_64-2.6.32-431.el6
-kernel.x86_64-2.6.32-431.23.3.el6
+libstdc++-devel.x86_64-4.4.7-4.el6
-puppet.noarch-3.7.0-1.el6
+puppet.noarch-3.7.1-1.el6

I compared the systems with rsync (from the working one):

rsync \
   --checksum -ani \
   --exclude log \
   --exclude /vagrant \
   --exclude /root \
   --exclude /proc \
   --exclude /sys \
   --exclude /selinux \
   --exclude /dev \
   --exclude /lib/modules \
   --exclude /usr/src \
   --exclude /boot \
   --delete / \
   root@broken:/ \
| grep -ve "^." \
| grep -vEie "puppet|vbox|virtualbox|var/lib/yum|/var/run|sss|var/lib/rpm|logs"

See results here: http://paste.ubuntu.com/8367930/

I can't figure out what of the changes that popped up might have an impact on a Java Spring application.

edit: Glimpse at the offending code added.

tomcat

1 Answers

  1. You are mixed up static and runtime contexts: method getMessageSource called during this initialization static variables. But the applicationContext is initialized after that. That's why you get NPE - applicationContext is not yet initialized.

    • 0