Home / server / Questions / 988423
Accepted
Kurt
Asked: 2019-10-18 07:46:55 +0800 CST

pfsense with only carp addresses

  • 772

I currently have two pfsense boxes with a standard carp setup as per the docs, it works great.

Two part question here:

  1. how does anything even work? My experience when accidentally having two ips on the same network has not been pleasant. So how does having a carp ip and a physical ip manage to coexist so harmoniously?

which feeds into the real question

  1. should the pfsense boxes only have carp addresses for wan & lan and all communication/routing/whatever happen over the real sync ip address and nic?

ie: I'm proposing that the wan and lan do not have ips, only the sync nic has an ip, there will be a carp wan ip and carp lan ip

pfsense

1 Answers

  1. Best Answer

    How does it even work

    carp is a protocol, pretty much like VRRP, sharing IPs among several hosts.

    If you were to run tcpdump on the interface on which your carp was configured, you would be able to see messages such as:

    # tcpdump -vvni vlan1000 proto carp
tcpdump: listening on vlan1000, link-type EN10MB
19:34:59.688730 carp 1.2.3.4 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=10 advskew=5 demote=0 [tos 0x4] (ttl 255, id 25329, len 56)

 tcpdump -vvni vlan42 proto carp   
 tcpdump: listening on vlan42, link-type EN10MB
  19:35:56.610612 carp 10.42.42.5 > 224.0.0.18: CARPv2-advertise 36: vhid=42 advbase=10 advskew=5 demote=0 (DF) [tos 0x10] (ttl 255, id 13021, len 56)

    Carp members would send messages to a multicast address, advertising their presence to each other. Only one member would be elected as a master on a given carp group.

    In case of active-active scenarios, you could still have two or more hosts serving the same carp address, assuming pf states are shared (pfsync) among carp members.

    Carp addresses would resolve to predictive hardware addresses, based on your carp VHID:

    # arp -na|grep 'carp.*permanent'
10.42.40.1                           00:00:5e:00:01:0b carp11 permanent  l
10.42.41.1                           00:00:5e:00:01:08  carp8 permanent  l
10.42.42.1                           00:00:5e:00:01:2a carp42 permanent  l
10.42.43.1                           00:00:5e:00:01:07  carp7 permanent  l
10.42.44.1                           00:00:5e:00:01:04  carp4 permanent  l
10.42.45.1                           00:00:5e:00:01:03  carp3 permanent  l
10.42.46.1                           00:00:5e:00:01:0a carp10 permanent  l
10.42.242.1                          00:00:5e:00:01:02  carp2 permanent  l
10.42.252.1                          00:00:5e:00:01:64 carp100 permanent  l
10.42.253.1                          00:00:5e:00:01:05  carp5 permanent  l
10.42.254.1                          00:00:5e:00:01:06  carp6 permanent  l
1.2.3.4                              00:00:5e:00:01:01  carp1 permanent  l
192.168.10.254                       00:00:5e:00:01:0c carp12 permanent  l

    For VHID 1, my carp address would have its mac set to 00:00:5e:00:01:01, for VHID 2 00:00:5e:00:01:02, ... and so on. Hence: regardless which is your carp master, clients in that network would not need to refresh their ARP cache, they would always contact the same hardware address.

    The reason why having two hosts sharing the same IPs in general is "not pleasant", is because for one: they don't have the same mac. Based on who replied to your last arp request, you can switch from one to the other unexpectedly. Also, both your pfSense share a similar configuration, ensure the same functions in your network. While when mistakenly re-attributing an IP, you usually have two completely different machines.

    Should CARP members also have physical IPs

    It depends. Most of the time: no. I tend to set one on management or public interfaces, making sure that I can still reach a backup host (eg: monitoring, troubleshooting some connectivity issue, ...), or just setting up pfSync.

    With FreeBSD, you may also be using hastd+ctld+ifstated sharing HA LUN devices. Then, I would also setup a physical IP on the main interface, as regardless of who's master, I'ld need hastd to sync my devices.

    • 1