Home / server / Questions / 1168619
Accepted
Bartosz Podemski
Asked: 2024-12-07 20:52:54 +0800 CST

Keepalived NAT with firewalld: STATE_INVALID_DROP blocking return packets from real servers

  • 772

I've set up Keepalived as a load balancer but am not using a Master/Standby configuration—just a single server.

The issue arises when firewalld is enabled. Without firewalld, everything works fine. However, when firewalld is running, packets returning from the real server are blocked with the STATE_INVALID_DROP rule.

The setup involves two interfaces: enp2s0 (clients zone) and enp3s0 (servers zone). Traffic flowing from enp2s0 to enp3s0 works correctly, but the problem occurs with traffic returning from enp3s0 to enp2s0.

Interestingly, if I use only firewalld with a route configured to reach the real server and remove Keepalived, the traffic flows without any issues. The problem only appears when both firewalld and Keepalived are in use together.

This is my zones config:

router1@router1:~$ sudo firewall-cmd --zone clients --list-all
clients (active)
  target: DROP
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: enp2s0
  sources:
  services: http
  ports:
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
router1@router1:~$ sudo firewall-cmd --zone servers --list-all
servers (active)
  target: DROP
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: enp3s0
  sources:
  services: http
  ports:
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

And this is my politics config:

router1@router1:~$ sudo firewall-cmd --policy fromCliToSrv --list-all
fromCliToSrv (active)
  priority: -1
  target: DROP
  ingress-zones: clients
  egress-zones: servers
  services: http
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
router1@router1:~$ sudo firewall-cmd --policy fromSrvToCli --list-all
fromSrvToCli (active)
  priority: -1
  target: DROP
  ingress-zones: servers
  egress-zones: clients
  services: http
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

And this is my keepalived config:

router1@router1:~$ sudo cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

virtual_server 172.20.241.20 80 {
        lb_algo wrr
        lb_kind NAT
        protocol TCP
        persistence_timeout 600
        persistence_granularity 255.255.255.0

        real_server 10.0.1.5 {
                weight 1
        }
        real_server 10.0.1.6 {
                weight 2
        }
        real_server 10.0.1.7 {
                weight 3
        }
}

I tried adding forward ports to clients zone like this:

router1@router1:~$ sudo firewall-cmd --permanent --zone clients --add-forward-port=port=80:proto=tcp:toaddr=10.0.1.5:toport=80
success
router1@router1:~$ sudo firewall-cmd --permanent --zone clients --add-forward-port=port=80:proto=tcp:toaddr=10.0.1.6:toport=80
success
router1@router1:~$ sudo firewall-cmd --permanent --zone clients --add-forward-port=port=80:proto=tcp:toaddr=10.0.1.7:toport=80
success
router1@router1:~$ sudo firewall-cmd --reload
success

But this config forwards all requests to 10.0.1.5 because it's first rule that was added

linux

1 Answers

  1. Best Answer

    This is because of how keepalived works in NAT mode.

    When using NAT in keepalived you are relying on netfilter (iptables) rules to mangle the packets in order to do load balancing (this is effectively layer 4 load balancing).

    FirewallD is also dictating traffic policy using netfilter (iptables) to block particular bits of traffic.

    My bet is that either FirewallD clobbers the keepalived netfilter config or vice versa, or the ordering and priority of the netfilter rules in both keepalived and FirewallD add up together in a manner that makes no sense.

    In any case your most pragmatic solution here is to not use FirewallD and directly make any network filtering policies using nft or iptables directly with a ruleset that won't interfere with keepalived.

    Or use load balancer software that just recreates the connections in the system itself, like haproxy.

    Id also make a point that the old virtual server load balancing modules used for keepalived (the ipvsadm stuff) aren't compatible with modern netfilter (nft based), at least last time I tried it.

    I have ultimately relegated keepalived in my headspace as a good VRRP implementation (failover) but poor load balancing solution.

    A common setup is to use keepalived for failover and haproxy for load balancing. This setup lets you have your cake and it it too, there should be plenty of stuff online on how to do that manually, even docker images and other pre-baked ready meal options if you are in a hurry.

    • 0