I have been using HAProxy for about two years with the same working configuration. I've installed HaProxy multiple times over the last week on Ubuntu 14.04, 15.10 and I am getting out of Memory crashes in HaProxy. It no longer works. I have attached the configuration I have been using as well as the errors I am seeing in syslog below.
I do have some ubuntu servers that I created a couple months ago that run the same version of HAProxy that do work with the exact configuration, but not sure the best way to pinpoint the issue.
Version information:
HA-Proxy version 1.5.14 2015/07/02
Copyright 2000-2015 Willy Tarreau <[email protected]>
You can substitute www.mywebsite.com with anything such as www.cnn.com if you want to try it.
haproxy.cfg
global
daemon
maxconn 256000
user haproxy
group haproxy
stats socket /var/run/haproxy.sock mode 0600 level admin
pidfile /var/run/haproxy.pid
spread-checks 5
defaults
maxconn 256000
log global
option redispatch
option allbackups
option abortonclose
option http-server-close
timeout connect 5000
timeout client 5000
timeout server 5000
timeout queue 5000
timeout check 5000
retries 2
# Statistics
listen stats :1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth myuser:mypass
# Front-end HTTP Rules
frontend f_http
bind *:80
mode http
default_backend b_http_us
# Frontend HTTPS Rules
frontend f_https
bind *:443
mode tcp
default_backend b_https_us
backend b_http_us
mode http
option accept-invalid-http-response
balance roundrobin
server www.mywebsite.com www.mywebsite.com:80 maxconn 100000 check inter 2s fastinter 2s downinter 2s
backend b_https_us
mode tcp
balance roundrobin
server www.mywebsite.com www.mywebsite.com:443 maxconn 100000 check inter 2s fastinter 2s downinter 2s
Here is my syslog
Mar 8 10:42:05 www kernel: [ 263.005325] gmain invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Mar 8 10:42:05 www kernel: [ 263.005331] gmain cpuset=/ mems_allowed=0
Mar 8 10:42:05 www kernel: [ 263.005340] CPU: 0 PID: 581 Comm: gmain Not tainted 4.2.0-27-generic #32-Ubuntu
Mar 8 10:42:05 www kernel: [ 263.005341] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Mar 8 10:42:05 www kernel: [ 263.005344] 0000000000000000 000000001c9253e8 ffff8800399bf958 ffffffff817eae99
Mar 8 10:42:05 www kernel: [ 263.005347] 0000000000000000 ffff88003da1e040 ffff8800399bf9e8 ffffffff817e8c8f
Mar 8 10:42:05 www kernel: [ 263.005349] ffffffff810c43f1 0000000000000000 0000000000000001 0000000000000000
Mar 8 10:42:05 www kernel: [ 263.005352] Call Trace:
Mar 8 10:42:05 www kernel: [ 263.005364] [<ffffffff817eae99>] dump_stack+0x45/0x57
Mar 8 10:42:05 www kernel: [ 263.005367] [<ffffffff817e8c8f>] dump_header+0x8c/0x214
Mar 8 10:42:05 www kernel: [ 263.005374] [<ffffffff810c43f1>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Mar 8 10:42:05 www kernel: [ 263.005381] [<ffffffff81185fa1>] oom_kill_process+0x1e1/0x3b0
Mar 8 10:42:05 www kernel: [ 263.005383] [<ffffffff81186726>] out_of_memory+0x556/0x5b0
Mar 8 10:42:05 www kernel: [ 263.005399] [<ffffffff8118c3c1>] __alloc_pages_nodemask+0x8c1/0xa10
Mar 8 10:42:05 www kernel: [ 263.005406] [<ffffffff811d37c1>] alloc_pages_current+0x91/0x100
Mar 8 10:42:05 www kernel: [ 263.005409] [<ffffffff81182510>] __page_cache_alloc+0xa0/0xc0
Mar 8 10:42:05 www kernel: [ 263.005419] [<ffffffff81211d90>] ? poll_select_copy_remaining+0x140/0x140
Mar 8 10:42:05 www kernel: [ 263.005422] [<ffffffff81184a4a>] filemap_fault+0x14a/0x3f0
Mar 8 10:42:05 www kernel: [ 263.005425] [<ffffffff81211d90>] ? poll_select_copy_remaining+0x140/0x140
Mar 8 10:42:05 www kernel: [ 263.005430] [<ffffffff811b0a80>] __do_fault+0x50/0xe0
Mar 8 10:42:05 www kernel: [ 263.005433] [<ffffffff811b5aab>] handle_mm_fault+0xf5b/0x17e0
Mar 8 10:42:05 www kernel: [ 263.005440] [<ffffffff81067a87>] __do_page_fault+0x197/0x400
Mar 8 10:42:05 www kernel: [ 263.005442] [<ffffffff81067d57>] trace_do_page_fault+0x37/0xa0
Mar 8 10:42:05 www kernel: [ 263.005446] [<ffffffff8105fe19>] do_async_page_fault+0x19/0x70
Mar 8 10:42:05 www kernel: [ 263.005451] [<ffffffff817f3bf8>] async_page_fault+0x28/0x30
Mar 8 10:42:05 www kernel: [ 263.005453] Mem-Info:
Mar 8 10:42:05 www kernel: [ 263.005457] active_anon:203766 inactive_anon:36 isolated_anon:0
Mar 8 10:42:05 www kernel: [ 263.005457] active_file:25 inactive_file:22 isolated_file:0
Mar 8 10:42:05 www kernel: [ 263.005457] unevictable:0 dirty:1 writeback:0 unstable:0
Mar 8 10:42:05 www kernel: [ 263.005457] slab_reclaimable:11917 slab_unreclaimable:32447
Mar 8 10:42:05 www kernel: [ 263.005457] mapped:31 shmem:138 pagetables:849 bounce:0
Mar 8 10:42:05 www kernel: [ 263.005457] free:1947 free_pcp:96 free_cma:0
Mar 8 10:42:05 www kernel: [ 263.005461] Node 0 DMA free:3960kB min:60kB low:72kB high:88kB active_anon:10040kB inactive_anon:4kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:20kB slab_reclaimable:432kB slab_unreclaimable:1188kB kernel_stack:0kB pagetables:96kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? yes
Mar 8 10:42:05 www kernel: [ 263.005466] lowmem_reserve[]: 0 975 975 975
Mar 8 10:42:05 www kernel: [ 263.005468] Node 0 DMA32 free:3828kB min:3828kB low:4784kB high:5740kB active_anon:805024kB inactive_anon:140kB active_file:100kB inactive_file:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032184kB managed:1000836kB mlocked:0kB dirty:4kB writeback:0kB mapped:120kB shmem:532kB slab_reclaimable:47236kB slab_unreclaimable:128600kB kernel_stack:1248kB pagetables:3300kB unstable:0kB bounce:0kB free_pcp:384kB local_pcp:384kB free_cma:0kB writeback_tmp:0kB pages_scanned:1116 all_unreclaimable? yes
Mar 8 10:42:05 www kernel: [ 263.005473] lowmem_reserve[]: 0 0 0 0
Mar 8 10:42:05 www kernel: [ 263.005475] Node 0 DMA: 2*4kB (EM) 11*8kB (UM) 4*16kB (UEM) 1*32kB (E) 1*64kB (U) 1*128kB (U) 0*256kB 1*512kB (E) 1*1024kB (E) 1*2048kB (M) 0*4096kB = 3968kB
Mar 8 10:42:05 www kernel: [ 263.005485] Node 0 DMA32: 9*4kB (UE) 2*8kB (EM) 66*16kB (M) 85*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3828kB
Mar 8 10:42:05 www kernel: [ 263.005493] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 8 10:42:05 www kernel: [ 263.005494] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 8 10:42:05 www kernel: [ 263.005495] 182 total pagecache pages
Mar 8 10:42:05 www kernel: [ 263.005497] 0 pages in swap cache
Mar 8 10:42:05 www kernel: [ 263.005499] Swap cache stats: add 0, delete 0, find 0/0
Mar 8 10:42:05 www kernel: [ 263.005499] Free swap = 0kB
Mar 8 10:42:05 www kernel: [ 263.005500] Total swap = 0kB
Mar 8 10:42:05 www kernel: [ 263.005501] 262044 pages RAM
Mar 8 10:42:05 www kernel: [ 263.005502] 0 pages HighMem/MovableOnly
Mar 8 10:42:05 www kernel: [ 263.005503] 7858 pages reserved
Mar 8 10:42:05 www kernel: [ 263.005503] 0 pages cma reserved
Mar 8 10:42:05 www kernel: [ 263.005504] 0 pages hwpoisoned
Mar 8 10:42:05 www kernel: [ 263.005505] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Mar 8 10:42:05 www kernel: [ 263.005510] [ 204] 0 204 8826 85 23 3 0 0 systemd-journal
Mar 8 10:42:05 www kernel: [ 263.005512] [ 210] 0 210 10945 148 24 3 0 -1000 systemd-udevd
Mar 8 10:42:05 www kernel: [ 263.005514] [ 334] 100 334 25074 64 20 3 0 0 systemd-timesyn
Mar 8 10:42:05 www kernel: [ 263.005516] [ 542] 0 542 7286 56 20 3 0 0 cron
Mar 8 10:42:05 www kernel: [ 263.005518] [ 564] 0 564 70809 187 43 3 0 0 accounts-daemon
Mar 8 10:42:05 www kernel: [ 263.005520] [ 575] 0 575 7155 77 18 3 0 0 systemd-logind
Mar 8 10:42:05 www kernel: [ 263.005521] [ 577] 104 577 64093 161 28 3 0 0 rsyslogd
Mar 8 10:42:05 www kernel: [ 263.005523] [ 585] 105 585 10712 112 26 3 0 -900 dbus-daemon
Mar 8 10:42:05 www kernel: [ 263.005534] [ 658] 0 658 17482 187 37 3 0 -1000 sshd
Mar 8 10:42:05 www kernel: [ 263.005537] [ 662] 0 662 4024 37 13 3 0 0 agetty
Mar 8 10:42:05 www kernel: [ 263.005539] [ 713] 0 713 1056 19 7 3 0 0 haproxy-systemd
Mar 8 10:42:05 www kernel: [ 263.005540] [ 716] 108 716 15492 1223 26 3 0 0 haproxy
Mar 8 10:42:05 www kernel: [ 263.005542] [ 717] 108 717 212748 200030 411 3 0 0 haproxy
Mar 8 10:42:05 www kernel: [ 263.005544] [ 718] 0 718 24943 244 49 3 0 0 sshd
Mar 8 10:42:05 www kernel: [ 263.005546] [ 720] 0 720 9184 202 23 3 0 0 systemd
Mar 8 10:42:05 www kernel: [ 263.005547] [ 721] 0 721 14719 419 33 3 0 0 (sd-pam)
Mar 8 10:42:05 www kernel: [ 263.005549] [ 740] 0 740 5615 402 15 3 0 0 bash
Mar 8 10:42:05 www kernel: [ 263.005551] [ 761] 0 761 1876 19 9 3 0 0 tail
Mar 8 10:42:05 www kernel: [ 263.005552] Out of memory: Kill process 717 (haproxy) score 788 or sacrifice child
Mar 8 10:42:05 www kernel: [ 263.005612] Killed process 717 (haproxy) total-vm:850992kB, anon-rss:800120kB, file-rss:0kB
Mar 8 10:42:06 www systemd[1]: haproxy.service: Service hold-off time over, scheduling restart.
Mar 8 10:42:06 www systemd[1]: Stopped HAProxy Load Balancer.
Mar 8 10:42:06 www systemd[1]: Starting HAProxy Load Balancer...
Mar 8 10:42:06 www systemd[1]: Started HAProxy Load Balancer.