we have a new(ish) WLS 9.2 installation
Over the last day or so, after restarting WLS, after a while, the admin server unexpectedly shuts down, preceded in the logs by many (hundreds) occurrences of exceptions like the below. This may be connected with having done an EAR redeployment, but if so the adverse effect seems to persist over a WLS restart.
We have tried restoring LDAP (from a backup months ago which we are pretty sure is OK)
It seems to shut down in a semi-orderly way after being told to do so by the JVM - see the other excerpt from the log below. Goolgling suggests that's becasue system.exit has been called, but there's no indication I can see in the log about what has caused that to happen (other than the stream of exceptions)
Any ideas on causes or fixes?
####<Aug 13, 2014 6:50:12 AM BST> <Critical> <EmbeddedLDAP> <csrpth01-omch> <AdminServer_9002> <[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default
(self-tuning)'> <<WLS Kernel>> <> <> <1407909012546> <000000> <java.lang.NullPointerException
at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:715)
at weblogic.socket.SocketMuxer.deliverEndOfStream(SocketMuxer.java:684)
at weblogic.ldap.MuxableSocketLDAP$LDAPSocket.close(MuxableSocketLDAP.java:118)
at com.octetstring.vde.Connection.close(Connection.java:166)
at com.octetstring.vde.WorkThread.executeWorkQueueItem(WorkThread.java:89)
at weblogic.ldap.LDAPExecuteRequest.run(LDAPExecuteRequest.java:50)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
And
####<Aug 13, 2014 7:06:23 AM BST> <Notice> <WebLogicServer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983789> <BEA-000388> <JVM
called WLS shutdown hook. The server will force shutdown now>
####<Aug 13, 2014 7:06:23 AM BST> <Alert> <WebLogicServer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983802> <BEA-000396> <Serv
er shutdown has been requested by <WLS Kernel>>
####<Aug 13, 2014 7:06:23 AM BST> <Notice> <WebLogicServer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983807> <BEA-000365> <Ser
ver state changed to FORCE_SUSPENDING>
####<Aug 13, 2014 7:06:23 AM BST> <Notice> <Server> <csrpth01-omch> <AdminServer_9002> <DynamicSSLListenThread[DefaultSecure]> <<WLS Kernel>> <> <> <14079099838
13> <BEA-002607> <Channel "DefaultSecure" listening on 10.204.1.232:7002 was shutdown.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <Deployer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983822> <BEA-149059> <Module hdm-
dashboard-admin.war of application hdm-dashboard-admin is transitioning from STATE_ACTIVE to STATE_ADMIN on server AdminServer_9002.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <Deployer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983830> <BEA-149060> <Module hdm-
dashboard-admin.war of application hdm-dashboard-admin successfully transitioned from STATE_ACTIVE to STATE_ADMIN on server AdminServer_9002.>
####<Aug 13, 2014 7:06:23 AM BST> <Notice> <WebLogicServer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983838> <BEA-000365> <Ser
ver state changed to ADMIN>
####<Aug 13, 2014 7:06:23 AM BST> <Notice> <WebLogicServer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983839> <BEA-000365> <Ser
ver state changed to FORCE_SHUTTING_DOWN>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <JMX> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983857> <BEA-149513> <JMX Connector Se
rver stopped at service:jmx:iiop://staginghdm.vfl.vodafone:9002/jndi/weblogic.management.mbeanservers.domainruntime .>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <Diagnostics> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983859> <BEA-320002> <The Diag
nostics subsystem is stopping on Server AdminServer_9002.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <JMX> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983925> <BEA-149513> <JMX Connector Se
rver stopped at service:jmx:iiop://staginghdm.vfl.vodafone:9002/jndi/weblogic.management.mbeanservers.edit .>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <JMX> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983926> <BEA-149513> <JMX Connector Se
rver stopped at service:jmx:iiop://staginghdm.vfl.vodafone:9002/jndi/weblogic.management.mbeanservers.runtime .>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <WebService> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983927> <BEA-220028> <Web Servi
ce reliable agents are suspended.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <WebService> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983937> <BEA-220029> <Web Servi
ce reliable agents are shut down.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <SAFService> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983943> <BEA-281004> <SAF Servi
ce has been suspended.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <SAFService> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983950> <BEA-281005> <SAF Servi
ce has been shut down.>
####<Aug 13, 2014 7:06:23 AM BST> <Info> <Deployer> <csrpth01-omch> <AdminServer_9002> <Thread-1> <<WLS Kernel>> <> <> <1407909983964> <BEA-149059> <Module hdm-
As rightly mentioned the server logs are not providing any useful information.
I would suggest you to enable weblogic debug flags related to life cycle events.
For such life cycle issue, I have posted related flags on my blog:
http://www.weblogic-tips.com/2014/08/16/debug-weblogic-startup-related-issues/
Try to enable the mentioned debug flags and update the thread with enhanced server log.
Thanks, weblogic-tips