I have a standard 2 node HAproxy load balancer setup with pacemaker 1.1 (using pcs) and haproxy 1.5. I have 2 floating IP addresses with a constraint with the haproxy service as a haproxy-clone resource. I need to add a new floating IP with the same constraint on the hparoxy-clone service. My first problem is when I create the floating IP resource it starts on the node without the other IP addresses. I then run the pcs constraint colocation add haproxy-clone with floatIP_189 (floatIP_189 is my new address). This doesn't move the IP address to the other node and the haproxy-clone resource stops. I can get the haproxy service started again by restarting the pacemaker service on both nodes.
How can I add a new floating IP to pacemaker without dropping traffic or otherwise disrupting the other sites on the load balancer.?
Below are the haproxy.cfg files, the output of the pcs status command and the output of the pcs config command;
[root@t-haproxylb3 haproxy]# cat haproxy.cfg
peers QAHAproxypeers
peer t-haproxylb3 10.x.x.185:1024
peer t-haproxylb4 10.x.x.186:1024
global
log 127.0.0.1 local0
# log /dev/log local0
# log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /var/lib/haproxy/stats
stats timeout 30s
tune.ssl.default-dh-param 2048
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
option redispatch
option forwardfor
option http-server-close
maxconn 5000
timeout connect 5s
timeout client 5h
timeout server 5h
timeout queue 30s
timeout http-request 5s
timeout http-keep-alive 15s
listen stats *:1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /haproxy_stats
stats auth admin:password
stats admin if TRUE
frontend http_in
bind *:80
###Add new acl and use_backend entry for each new site
###new backend sections will be needed as well
acl is_clients hdr(host) -i clients.qa.racingcars.com
acl is_apps hdr(host) -i apps.qa.racingcars.com
acl is_dad hdr(host) -i dad.qa.racingcars.com
acl is_scripting hdr(host) -i scripting.qa.racingcars.com
acl is_racingcarsnet hdr_end(host) -i racingcars.net
use_backend http_client if is_clients
use_backend http_apps if is_apps
use_backend http_dad if is_dad
use_backend http_scripting if is_scripting
use_backend http_racingcarsnet if is_racingcarsnet
option forwardfor
option http-server-close
frontend https_in_ssl_apps
bind 10.x.x.187:443 ssl crt /etc/ssl/private/apps.racingcars.com.pem
mode http
use_backend https_ssl_apps
option forwardfor
option http-server-close
frontend https_in_ssl_clients
bind 10.x.x.188:443 ssl crt /etc/ssl/private/clients.racingcars.com.pem
mode http
use_backend https_ssl_clients
option forwardfor
option http-server-close
frontend https_in_ssl_scripting
bind 10.x.x.189:443 ssl crt /etc/ssl/private/clients.racingcars.com.pem
mode http
use_backend https_ssl_scripting
option forwardfor
option http-server-close
frontend https_in_ssl
mode http
# bind *:443 ssl crt /etc/ssl/private/ no-sslv3
bind *:443 ssl crt /etc/ssl/private/
reqadd X-Forwarded-Proto:\ https
####### commented out below to enable https pass-through for apps
# use_backend http_clients if { ssl_fc_sni clients.racingcars.com }
# use_backend http_apps if { ssl_fc_sni apps.racingcars.com }
acl is_ssl_racingcarsnet hdr_end(host) -i racingcars.net
use_backend http_racingcarsnet if is_ssl_racingcarsnet
backend http_clients
balance source
cookie SRV_ID prefix
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
###This site does not use host header - only the page name is needed###
# option httpchk HEAD /Default.aspx
###Added host header so haproxy can route around NLB - use below for checking###
option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ clients.qa.racingcars.com
server websvr03 10.x.x.183:80 cookie web3 weight 5 check
# server websvr04 10.x.x.118:80 cookie web4 weight 5 check
backend https_ssl_clients
balance source
cookie SRV_ID prefix
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ clients.qa.racingcars.com
server websvr03 10.x.x.183:443 cookie web3 weight 5 check ssl verify none
# server websvr04 10.x.x.118:443 cookie web4 weight 5 check ssl verify none
backend http_apps
balance roundrobin
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
###This site uses host headers so this type of check is required###
option httpchk HEAD /default.htm HTTP/1.1\r\nHost:\ apps.qa.racingcars.com
server websvr03 10.x.x.182:80 cookie web3 weight 5 check
# server websvr04 10.x.x.116:80 cookie web4 weight 5 check
backend https_ssl_apps
balance roundrobin
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
###This site uses host headers so this type of check is required###
option httpchk HEAD /default.htm HTTP/1.1\r\nHost:\ apps.qa.racingcars.com
server websvr03 10.x.x.182:443 cookie web3 weight 5 check ssl verify none
# server websvr04 10.x.x.116:443 cookie web4 weight 5 check ssl verify none
backend http_dad
balance roundrobin
cookie SRV_ID prefix
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
###This site does not use host header - only the page name is needed###
option httpchk HEAD /login.aspx HTTP/1.1\r\nHost:\ dad.qa.racingcars.com
server websvr03 10.x.x.182:80 cookie web3 weight 5 check
# server websvr04 10.x.x.116:80 cookie web4 weight 5 check
backend http_scripting
balance roundrobin
cookie SRV_ID prefix
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
###This site uses host header so this type of check is required###
option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ scripting.qa.racingcars.com
server websvr03 10.x.x.184:80 cookie web3 weight 5 check
# server websvr04 10.x.x.116:80 cookie web4 weight 5 check
backend https_ssl_scripting
balance source
cookie SRV_ID prefix
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
option httpchk HEAD /Default.aspx HTTP/1.1\r\nHost:\ scripting.qa.racingcars.com
server websvr03 10.x.x.184:443 cookie web3 weight 5 check ssl verify none
# server websvr04 10.x.x.118:443 cookie web4 weight 5 check ssl verify none
backend http_racingcarsnet
balance roundrobin
cookie SRV_ID prefix
stick-table type ip size 1m expire 6h peers QAHAproxypeers
stick on src
###This site uses host header so this type of check is required###
option httpchk HEAD /default.aspx HTTP/1.1\r\nHost:\ test.racingcars.net
# server websvr03 10.x.x.115:80 cookie web3 weight 5 check
# server websvr04 10.x.x.117:80 cookie web4 weight 5 check
[root@t-haproxylb3 haproxy]# pcs status
Cluster name: testcluster2
Stack: corosync
Current DC: t-haproxylb3 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Tue Dec 20 16:55:37 2016 Last change: Tue Dec 20 14:15:59 2016 by root via cibadmin on t-haproxylb3
2 nodes and 5 resources configured
Online: [ t-haproxylb3 t-haproxylb4 ]
Full list of resources:
Clone Set: haproxy-clone [haproxy]
Started: [ t-haproxylb3 ]
Stopped: [ t-haproxylb4 ]
floatIP_187 (ocf::heartbeat:IPaddr2): Started t-haproxylb3
floatIP_188 (ocf::heartbeat:IPaddr2): Started t-haproxylb3
floatIP_189 (ocf::heartbeat:IPaddr2): Started t-haproxylb3
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@t-haproxylb3 haproxy]# pcs config
Cluster Name: testcluster2
Corosync Nodes:
t-haproxylb3 t-haproxylb4
Pacemaker Nodes:
t-haproxylb3 t-haproxylb4
Resources:
Clone: haproxy-clone
Resource: haproxy (class=systemd type=haproxy)
Operations: monitor interval=10s (haproxy-monitor-interval-10s)
Resource: floatIP_187 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.x.x.187 cidr_netmask=32
Operations: start interval=0s timeout=20s (floatIP_187-start-interval-0s)
stop interval=0s timeout=20s (floatIP_187-stop-interval-0s)
monitor interval=30s (floatIP_187-monitor-interval-30s)
Resource: floatIP_188 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.x.x.188 cidr_netmask=32
Operations: start interval=0s timeout=20s (floatIP_188-start-interval-0s)
stop interval=0s timeout=20s (floatIP_188-stop-interval-0s)
monitor interval=30s (floatIP_188-monitor-interval-30s)
Resource: floatIP_189 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.x.x.189 cidr_netmask=32
Operations: start interval=0s timeout=20s (floatIP_189-start-interval-0s)
stop interval=0s timeout=20s (floatIP_189-stop-interval-0s)
monitor interval=30s (floatIP_189-monitor-interval-30s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Ordering Constraints:
Colocation Constraints:
haproxy-clone with floatIP_187 (score:INFINITY) (id:colocation-haproxy-clone-f loatIP_187-INFINITY)
haproxy-clone with floatIP_188 (score:INFINITY) (id:colocation-haproxy-clone-f loatIP_188-INFINITY)
haproxy-clone with floatIP_189 (score:INFINITY) (id:colocation-haproxy-clone-f loatIP_189-INFINITY)
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
resource-stickiness: 100
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: testcluster2
dc-version: 1.1.15-11.el7_3.2-e174ec8
have-watchdog: false
stonith-enabled: false
Quorum:
Options:
You should be working on an offline copy of your cib (pacemaker config), and then pushing it into the cluster only when you're ready. This should be what you want to do:
Pull the configuration out of the cluster into an xml file:
Then add your virtual IP resource to the file:
Add the constraints to the file:
NOTE: You'll want to add the ordering constraint for all of them, right?
If you add the IP address to the appropriate interface manually before pushing your new CIB (next step), Pacemaker should probe the system to determine where things are and won't need to take any action (won't stop haproxy) before starting the new VIP.
Verify you didn't goof something up, and push it into the cluster:
I am not sure how you'd kick ha-proxy into binding to the address; maybe there is a "reload" or "adjust" type of command. Hope that helps.