Questions[max-file-descriptors](server)

Our server recently ran out of file descriptors, and in regards to that I have some questions. ulimit -n is supposed to give me the maximum number of open file descriptors. That number is 1024. I checked the number of open file descriptors by running lsof -u root |wc -l and got 2500 fds. That is a lot more than 1024, so I guessed that would mean the number 1024 is per process, not per user, as I though. Well, I ran lsof -p$PidOfGlassfish|wc -l and got 1300. This is the part I don't get. If ulimit -n is not the maximum number of processes per user or per process, then what is it good for? Does it not apply to the root user? And if so, how could I then get the error messages about running out of file descriptor?

EDIT: The only way I can make sense out of ulimit -n is if it applies the the number of open files (as stated in the bash manual) rather than the number of file handles (different processes can open the same file). If this is the case, then simply listing the number of open files (grepping on '/', thus excluding memory mapped files) is not sufficent:

lsof -u root |grep /|sort  -k9  |wc -l #prints '1738'

To actually see the number of open files, I would need to filter on the name column on only print the unique entries. Thus the following is probably more correct:

lsof -u root |grep /|sort  -k9 -u |wc -l #prints '604'

The command above expects output on the following format from lsof:

java      32008 root  mem       REG                8,2 11942368      72721 /usr/lib64/locale/locale-archive
vmtoolsd   4764 root  mem       REG                8,2    18624     106432 /usr/lib64/open-vm-tools/plugins/vmsvc/libguestInfo.so

This at least gives me number less than 1024 (the number reported by ulimit -n), so this seems like a step in the right direction. "Unfortunately" I am not experiencing any problems with running out of file descriptors, so I will have a hard time validating this.

How can lsof report more open files than what ulimit says is the limit?

prod_web3(i-ca0b05aa):~$ sudo lsof | wc -l
4399
prod_web3(i-ca0b05aa):~$ ulimit -n
1024

Server applications running on linux often require large quantities of open file handlers, for ex. HBase ulimit, Hadoop epoll limit

This wiki entry should serve as documentary for Linux file limits configuration.

• What is soft vs. hard limit?
• How to control the hard limit?
• How to control the soft limit?
• Does kernel fs.file-max and user ulimit -n related?

Please describe the Linux distribution under which your configuration is valid as various vendors configure stuff differently.

Update Based on lstvan answer:

For people looking to automate this, at least on Ubuntu servers you can put this in your machine installation scripts:

echo 'fs.file-max = 65000' > /etc/sysctl.d/60-file-max.conf
echo '* soft nofile 65000' > /etc/security/limits.d/60-nofile-limit.conf
echo '* hard nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf
echo 'root soft nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf
echo 'root hard nofile 65000' >> /etc/security/limits.d/60-nofile-limit.conf

I have a java process (Glassfish) which is leaking file descriptors. I know this because I get the helpful java.io.IOException: Too many open files exception. I can look in /proc/PID#/fd and see all the open file descriptors. When I use lsof I get a very large number of entries like this:

java 18510 root 8811u sock 0,4 1576079 can't identify protocol
java 18510 root 8812u sock 0,4 1576111 can't identify protocol
java 18510 root 8813u sock 0,4 1576150 can't identify protocol

I see 12 new ones created per minute. What options can I use on lsof or what other tools are available to me to help track down socket file descriptors where the protocol can't be identified?

We recently began load testing our application and noticed that it ran out of file descriptors after about 24 hours.

We are running RHEL 5 on a Dell 1955:

CPU: 2 x Dual Core 2.66GHz 4MB 5150 / 1333FSB RAM: 8GB RAM HDD: 2 x 160GB 2.5" SATA Hard Drives

I checked the file descriptor limit and it was set at 1024. Considering that our application could potentially have about 1000 incoming connections as well as a 1000 outgoing connections, this seems quite low. Not to mention any actual files that need to be opened.

My first thought was to just increase the ulimit -n parameter by a few orders of magnitude and then re-run the test but I wanted to know any potential ramifications of setting this variable too high.

Are there any best practices towards setting this other than figuring out how many file descriptors our software can theoretically open?