Home / server / Questions / 271691
Accepted
Hoon
Asked: 2011-05-20 13:42:11 +0800 CST

"Too many openfiles" error with Glassfish on Ubuntu

  • 772

I am having issues with tuning Glassfish 2.1.1 on Ubuntu 10.04 LTS 64-bit (Amazon EC2). As soon as I change the HTTP Service thread count from the default 5 value to 100, I get the following errors in the server.log.

java.net.SocketException: Too many open files

Other errors include:

[#|2011-05-19T15:41:38.034-0500|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=16;_ThreadName=Timer-20;_RequestID=1bd7cd3e-0011-4ebc-95e5-487b96c76b20;|"DPL8011: autodeployment failure while deploying the application : null"|#]

[#|2011-05-19T15:41:39.555-0500|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=Timer-1;_RequestID=0d9630b5-2752-4ffb-ac7c-1cf51920155a;|
java.lang.NullPointerException
    at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
    at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
    at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
    at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:967)
    at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
    at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
    at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
    at javax.management.timer.Timer.sendNotification(Timer.java:1237)
    at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
    at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
|#]

and 

[#|2011-05-19T16:30:40.228-0500|SEVERE|sun-appserver2.1|org.apache.jasper.servlet.JspServlet|_ThreadID=16;_ThreadName=httpWorkerThread-4949-48;_RequestID=63d6908e-cc09-4fa8-aac0-241e7582c42f;|PWC6117: File "/opt/glassfish-v2.1.1-b31g/lib/install/applications/admingui/adminGUI_war/header.jsp" not found|#]

and 

[#|2011-05-19T16:30:40.229-0500|SEVERE|sun-appserver2.1|org.apache.jasper.servlet.JspServlet|_ThreadID=17;_ThreadName=httpWorkerThread-4949-46;_RequestID=869579eb-887d-4dc4-b0fc-edc4e41755a7;|PWC6117: File "/opt/glassfish-v2.1.1-b31g/lib/install/applications/admingui/adminGUI_war/homePage.jsp" not found|#]

Googling I found the follow resources:

http://felipeferreira.net/?p=873 http://www.netadmintools.com/art295.html

My /etc/security/limits.confg has the following configuration. And I have change the tcp settings as noted in http://mariosgaee.blogspot.com/2011/04/glassfish-211-on-linux-performance.html

*       soft    nofile          65535
*       hard    nofile          65535
*       soft    stack           unlimited
*       hard    stack           unlimited

/proc/sys/fs/file-max has a value of 762655 (I did not change this) but I did add to /etc/sysctl.conf as 'fs.file-max = 762655'

ulimit output

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Any ideas what may be causing this issue? Thanks in advance!

ubuntu glassfish

3 Answers

  1. Best Answer

    To see actual limits of running process you may use /proc , just get pid of your glassfish/java process and look at cat /proc/$PID_OF_PROCESS/limits It should be "Max open files" there. Also you can monitor number of open files with "lsof -p".

    • 1

  2. Continuing the previous answer: 1. Did you restart your glassfish server after changing limits.conf? 2. What used does glassfish run under? Do it's limits match what you set in limits.conf? Sometimes startup scripts will call ulimit for you, overriding what you set in limits.conf 3. You mention your run Ubuntu 10.04. Check and see if your pam setup load pam_limits.so, otherwise your limits.conf will be ignored

    • 0

  3. Try to add the line in /etc/pam.d/common-session : 

    session required pam_limits.so

    and reboot can be required.

    • 0