Questions[ulimit](server)

I have two CentOS 5 servers with nearly identical specs. When I login and do ulimit -u, on one machine I get unlimited, and on the other I get 77824.

When I run a cron like:

* * * * * ulimit -u > ulimit.txt

I get the same results (unlimited, 77824).

I am trying to determine where these are set so that I can alter them. They are not set in any of my profiles (.bashrc, /etc/profile, etc.). These wouldn't affect cron anyway) nor in /etc/security/limits.conf (which is empty).

I have scoured google and even gone so far as to do grep -Ir 77824 /, but nothing has turned up so far. I don't understand how these machines could have come preset with different limits.

I am actually wondering not for these machines, but for a different (CentOS 6) machine which has a limit of 1024, which is far too small. I need to run cron jobs with a higher limit and the only way I know how to set that is in the cron job itself. That's ok, but I'd rather set it system wide so it's not as hacky.

Thanks for any help. This seems like it should be easy (NOT).

EDIT -- SOLVED

Ok, I figured this out. It seems to be an issue either with CentOS 6 or perhaps my machine configuration. On the CentOS 5 configuration, I can set in /etc/security/limits.conf:

* - nproc unlimited

and that would effectively update the accounts and cron limits. However, this does not work in my CentOS 6 box. Instead, I must do:

myname1 - nproc unlimited
myname2 - nproc unlimited
...

And things work as expected. Maybe the UID specification works to, but the wildcard (*) definitely DOES NOT here. Oddly, wildcards DO work for the nofile limit.

I still would love to know where the default values are actually coming from, because by default, this file is empty and I couldn't see why I had different defaults for the two CentOS boxes, which had identical hardware and were from the same provider.

Can anyone explain in layman's terms what the difference between soft and hard limit is?

Should I set my soft and hard limit to be the same? Or should soft be significantly lower? Does the system benefit either way?

I have a long running process that is eventually going to hit the max open file limit. I know how to change that after it fails, but is there a way to change that for the running process, from the command line?

I notice that on a new CentOS image that I just booted up off of EC2 that the ulimit default is 1024 open files, but /proc/sys/fs/file-max is set at 761,408 and I'm wondering how these two limits work together. I'm guessing that ulimit -n is a per-user limit of number of file descriptors while /proc/sys/fs/file-max is system-wide? If that's the case, say I've logged in twice as the same user -- does each logged-in user have a 1024 limit on number of open files, or is it a limit of 1024 combined open files between each of those logged-in users?

And is there much performance impact to setting your max file descriptors to a very high number, if your system isn't ever opening very many files?

We recently began load testing our application and noticed that it ran out of file descriptors after about 24 hours.

We are running RHEL 5 on a Dell 1955:

CPU: 2 x Dual Core 2.66GHz 4MB 5150 / 1333FSB RAM: 8GB RAM HDD: 2 x 160GB 2.5" SATA Hard Drives

I checked the file descriptor limit and it was set at 1024. Considering that our application could potentially have about 1000 incoming connections as well as a 1000 outgoing connections, this seems quite low. Not to mention any actual files that need to be opened.

My first thought was to just increase the ulimit -n parameter by a few orders of magnitude and then re-run the test but I wanted to know any potential ramifications of setting this variable too high.

Are there any best practices towards setting this other than figuring out how many file descriptors our software can theoretically open?