I have a (probably quite old) CentOS 4.5 server with a custom java application running inside.
I found the application was crashing after some running time and found it was handling 1024 connections and trying to open one more socket when it died.
As a matter of fact if I check ulimit -n I can confirm it is 1024, so the application is getting closed since it has no more free file descriptors..
What bothers me is that there are hundreds of apparently inactive connections, in a "ESTABLISHED off" state, all from a relatively small number of IPs (about 200), and that they tend to add up as time goes by and clients connect, much like these which I see running netstat -nato:
tcp 0 0 ::ffff:10.39.151.20:10000 ::ffff:78.152.97.98:12059 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:10.39.151.20:10000 ::ffff:78.152.97.98:49179 ESTABLISHED off (0.00/0/0)
tcp 0 0 ::ffff:10.39.151.20:10000 ::ffff:78.152.97.42:45907 ESTABLISHED off (0.00/0/0)
I know it is not a DOS attack, the connections are legitimate, but the seem not to close after the clients connect and do a short data exchange with the server.. furthermore the pace is slow, since the are generated by 200 clients (counting different IP)..
Should I investigate on some weird application bug (maybe on jre 1.6), or dig into CentOS network configuration? I have no clue on what more to look upon..
Thanks in advance, any hint is appreciated!