Home / server / Questions / 818444
Accepted
Lucas Neves
Asked: 2016-12-03 11:46:57 +0800 CST

TCP out of memory

  • 772

I'm running one gcloud g1-small instance with Debian to power one Node server and one Apache server. I use Node Bouncy to redirect http requests to Node or Apache depending on req.headers.host. Each of my two websites are low-traffic, below 10k accesses a day.

After about two weeks running ininterruptly, both my Apache and Node websites stopped responding. I couldn't connect to the instance via SSH either. After rebooting and examining the logs, I found the following:

/var/log/kern.log:
kernel: [timestamp] TCP: out of memory -- consider tuning tcp_mem
(multiple times)

/var/log/apache2/error.log:
[Timestamp] [core:warn] [pid 573] (105)No buffer space available: AH00056: connect to listener on 0.0.0.0:8001
(multiple times)

There was nothing in my Node log file.

How can I prevent this error from happening again?

tcp apache-2.4

2 Answers

  1. Best Answer

    I have increased some TCP values, allowing for more memory to be used. I have added these lines to the server's startup script:

    sysctl -w net.ipv4.tcp_mem='116730 155640 233460'
sysctl -w net.ipv4.tcp_max_orphans='24576'

    While this shouldn't solve the problem forever, it should buy me more uptime before the server crashes. I've been rebooting the server once a month and had no problems so far.

    The real solution, though, would be fixing the memory leak that causes this problem in the first place.

    • 1

  2. Recently we experienced an interesting production problem. This application was running on multiple AWS EC2 instances behind Elastic Load Balancer. The application was running on GNU/Linux OS, Java 8, Tomcat 8 application server. All of sudden one of the application instances became unresponsive. All other application instances were handling the traffic properly. Whenever the HTTP request was sent to this application instance from the browser, we were getting following response to be printed on the browser.

    Proxy Error

    The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /.

    Reason: Error reading from remote server

    Let us see how we resolved this issue by assigning values for these properties in the server:

    net.core.netdev_max_backlog=30000 net.core.rmem_max=134217728 net.core.wmem_max=134217728 net.ipv4.tcp_max_syn_backlog=8192 net.ipv4.tcp_rmem=4096 87380 67108864 net.ipv4.tcp_wmem=4096 87380 67108864

    TCP: out of memory — consider tuning tcp_mem

    • -1