Our production admin server (WebLogic 10.3.5 running on Solaris 10) crashes from time to time. Logs show tons of these errors (several each minute):
<1-Jun-2012 2:28:34 o'clock AM EDT> <Critical> <EmbeddedLDAP> <BEA-000000> <java.lang.NullPointerException
at weblogic.socket.DevPollSocketMuxer.cleanupSocket(DevPollSocketMuxer.java:150)
at weblogic.socket.DevPollSocketMuxer.cancelIo(DevPollSocketMuxer.java:166)
at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:836)
at weblogic.socket.SocketMuxer.deliverEndOfStream(SocketMuxer.java:760)
at weblogic.ldap.MuxableSocketLDAP$LDAPSocket.close(MuxableSocketLDAP.java:128)
at com.octetstring.vde.Connection.close(Connection.java:166)
at com.octetstring.vde.WorkThread.executeWorkQueueItem(WorkThread.java:89)
at weblogic.ldap.LDAPExecuteRequest.run(LDAPExecuteRequest.java:50)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Eventually, the admin server runs out of memory:
<1-Jun-2012 12:29:59 o'clock PM EDT> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: GC overhead limit exceeded.
One does not necessarily cause the other, but it seems like a pretty good fit.
When inspecting the WebLogic code, we see this:
void cleanupSocket(MuxableSocket paramMuxableSocket, SocketInfo paramSocketInfo) {
this.sockRecords[paramSocketInfo.getFD()] = null; // DevPollSocketMuxer.java:150
super.cleanupSocket(paramMuxableSocket, paramSocketInfo);
}
protected void cancelIo(MuxableSocket paramMuxableSocket)
{
super.cancelIo(paramMuxableSocket);
cleanupSocket(paramMuxableSocket, paramMuxableSocket.getSocketInfo()); // DevPollSocketMuxer.java:166
}
So paramMuxableSocket.getSocketInfo() would be null. I'm at a loss for explaining this... Anyone have an idea?
Thanks!
We ended up boosting the max heap size setting to 512 MB and reducing the number of calls to the AdminServer LDAP by caching results on the client side. Problem solved.