Questions[unicast](server)

I would like to use IPv6 on my LAN and would like to be able to go out to the WAN as well. My network is a classic IPv4 environment. It works well.

My ISP provides Global Unicast IPv6 address on the WAN interface of my router (pfsense 2.5.2). I can ping google's IPv6 address from my router:

PING6(56=40+8+8 bytes) <my-global-unicast-iv6-address> --> 2a00:1450:400d:806::200e
16 bytes from 2a00:1450:400d:806::200e, icmp_seq=0 hlim=118 time=17.492 ms
16 bytes from 2a00:1450:400d:806::200e, icmp_seq=1 hlim=118 time=12.482 ms
16 bytes from 2a00:1450:400d:806::200e, icmp_seq=2 hlim=118 time=12.134 ms
16 bytes from 2a00:1450:400d:806::200e, icmp_seq=3 hlim=118 time=11.707 ms
16 bytes from 2a00:1450:400d:806::200e, icmp_seq=4 hlim=118 time=21.414 ms

--- 2a00:1450:400d:806::200e ping6 statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 11.707/15.046/21.414/3.814 ms

Behind the router the same thing does not work. I cannot ping google's IPv6 address. I need help to configure the LAN side properly.

May I ask your assistance please ?

EDIT: I had a call with my ISP and confirmed that ping might not work from my LAN. Maybe the the contact was not the right person to answer all my question because he answered that he does not know why but ping should not work but everything else should. Weird.

What I am trying to do?

I'm trying to acquire 3 public IP addresses via DHCP on a single physical ISP upstream cable.

What goes wrong?

Renewals go kind of wrong. From interfaces, virtual0 works just fine when it tries to renew via unicast connection to DHCP server. virtual1 and virtual2 interfaces fail with unicast. They drop into fallback mode to use broadcasts and succeed with them, but spam the logs full of stuff which cannot be good. I'm running Debian 8 and isc-dhcp-client v. 4.3.1 .

I set up the interfaces on boot like this:

IF_VIRTUAL_BASE=eth0
IF_PUB0=virtual0
IF_PUB1=virtual1
IF_PUB2=virtual2
IF_LAN=eth2

ifconfig "$IF_VIRTUAL_BASE" up
ip link add link "$IF_VIRTUAL_BASE" address 00:90:0b:ff:10:5b "$IF_PUB0" type macvlan
ip link set "$IF_PUB0" up
ip link add link "$IF_VIRTUAL_BASE" address 00:90:0b:ff:11:5b "$IF_PUB1" type macvlan
ip link set "$IF_PUB1" up
ip link add link "$IF_VIRTUAL_BASE" address 00:90:0b:ff:12:5b "$IF_PUB2" type macvlan
ip link set "$IF_PUB2" up

# Enable forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

I also do unset new_routers when completing dhcprequest for virtual1 and virtual2.

And guess what, this monstrosity kind of works:

Jul 11 20:45:43 gw dhclient: DHCPREQUEST on virtual0 to 255.255.255.255 port 67
Jul 11 20:45:43 gw dhclient: DHCPREQUEST on virtual1 to 255.255.255.255 port 67
Jul 11 20:45:43 gw ifup[582]: DHCPREQUEST on virtual0 to 255.255.255.255 port 67
Jul 11 20:45:43 gw ifup[592]: DHCPREQUEST on virtual1 to 255.255.255.255 port 67
Jul 11 20:45:43 gw dhclient: DHCPREQUEST on virtual2 to 255.255.255.255 port 67
Jul 11 20:45:43 gw ifup[634]: DHCPREQUEST on virtual2 to 255.255.255.255 port 67
Jul 11 20:45:46 gw dhclient: DHCPREQUEST on virtual1 to 255.255.255.255 port 67
Jul 11 20:45:46 gw ifup[592]: DHCPREQUEST on virtual1 to 255.255.255.255 port 67
Jul 11 20:45:46 gw dhclient: DHCPACK from 88.113.75.2
Jul 11 20:45:46 gw ifup[592]: DHCPACK from 88.113.75.2
Jul 11 20:45:46 gw logger: virtual1 (REBOOT): IP:  -> 88.113.75.59; GW:  -> 88.113.75.1

Jul 11 20:45:47 gw dhclient: DHCPREQUEST on virtual0 to 255.255.255.255 port 67
Jul 11 20:45:47 gw ifup[582]: DHCPREQUEST on virtual0 to 255.255.255.255 port 67
Jul 11 20:45:47 gw dhclient: DHCPACK from 88.113.75.2
Jul 11 20:45:47 gw ifup[582]: DHCPACK from 88.113.75.2
Jul 11 20:45:47 gw logger: virtual0 (REBOOT): IP:  -> 88.113.75.65; GW:  -> 88.113.75.1

Jul 11 20:45:50 gw dhclient: DHCPREQUEST on virtual2 to 255.255.255.255 port 67
Jul 11 20:45:50 gw ifup[634]: DHCPREQUEST on virtual2 to 255.255.255.255 port 67
Jul 11 20:45:50 gw dhclient: DHCPACK from 88.113.75.2
Jul 11 20:45:50 gw ifup[634]: DHCPACK from 88.113.75.2
Jul 11 20:45:50 gw logger: virtual2 (REBOOT): IP:  -> 88.113.75.61; GW:  -> 88.113.75.1

So they worked, right? And they are broadcast requests. Ok, fast forward an hour or so we gets looots of this:

Jul 11 21:45:09 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:45:11 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:45:22 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:45:25 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:45:32 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:45:43 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:45:46 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:45:56 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:46:05 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:46:11 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:46:21 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:46:30 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:46:33 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:46:42 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:46:47 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:47:00 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 21:47:01 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 21:47:09 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67

The virtual1 and virtual2 interfaces desperately try to renew their IPs via unicast connection to previous DHCP server 195.74.6.55 . They fail for the unicast. But something funny happens next, in the end they switch to broadcast as fallback and succeed!

Jul 11 22:30:41 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
Jul 11 22:30:47 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67
Jul 11 22:30:52 gw dhclient: DHCPREQUEST on virtual1 to 255.255.255.255 port 67
Jul 11 22:30:52 gw dhclient: DHCPACK from 88.113.75.2
Jul 11 22:30:52 gw logger: virtual1 (RENEW): IP: 88.113.75.59 -> 88.113.75.59; GW: 88.113.75.1 -> 88.113.75.1

Jul 11 22:30:58 gw dhclient: DHCPREQUEST on virtual2 to 255.255.255.255 port 67
Jul 11 22:30:58 gw dhclient: DHCPACK from 88.113.75.2
Jul 11 22:30:59 gw logger: virtual2 (RENEW): IP: 88.113.75.61 -> 88.113.75.61; GW: 88.113.75.1 -> 88.113.75.1

Observe the virtual0 interface:

Jul 11 22:38:17 gw dhclient: DHCPREQUEST on virtual0 to 195.74.6.55 port 67
Jul 11 22:38:18 gw dhclient: DHCPACK from 195.74.6.55
Jul 11 22:38:18 gw logger: virtual0 (RENEW): IP: 88.113.75.65 -> 88.113.75.65; GW: 88.113.75.1 -> 88.113.75.1

Conclusion is that for virtual0 interface, unicast (dhclient) DHCP requests work and for virtual1 and virtual2 only broadcast DHCP renewals work. So, it must be a routing problem, right? Here is what ip route shows after a typical boot:

root@gw:~# ip route
default via 88.113.75.1 dev virtual0
88.113.75.0/24 dev virtual0  proto kernel  scope link  src 88.113.75.65
88.113.75.0/24 dev virtual1  proto kernel  scope link  src 88.113.75.59
88.113.75.0/24 dev virtual2  proto kernel  scope link  src 88.113.75.61
172.16.8.0/28 via 172.16.8.2 dev tun0 # For openvpn
172.16.8.0/24 dev eth2  proto kernel  scope link  src 172.16.8.254 # For LAN
172.16.8.2 dev tun0  proto kernel  scope link  src 172.16.8.1 #For openvpn

How do I make the dhclients on macvlan-based interfaces virtual1 and virtual2 route their unicast renewal requests correctly and receive the responses appropriately?

I have done tons of Google searches, tried to flip-flop many settings, including firewall policies, policy-based routings, eth0 promisc mode and sysctl variables, stripping down the generic networking setup etc. I'm currently doing strace on dhclient. One combination of these settings must work. If more information is needed, I'm more than happy to provide.

EDIT1: dhclient strace became ready.

This happens in the very beginning:

bind(5, {sa_family=AF_PACKET, proto=0x7669, if1635087474, pkttype=PACKET_HOST, addr(0)={12652, }, 16) = 0
bind(6, {sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
sendto(5, "......", 342, 0, {sa_family=AF_PACKET, proto=0x7669, if1635087474, pkttype=PACKET_HOST, addr(0)={12652, }, 18) = 342

The "sendto(5" was, to my understanding, the first successful broadcast-based dhclient send.

Then, later the first failing unicast send:

sendto(6, "......", 300, 0, {sa_family=AF_INET, sin_port=htons(67), sin_addr=inet_addr("195.74.6.55")}, 16) = 300

And the next successful broadcast send:

sendto(5, "......", 342, 0, {sa_family=AF_PACKET, proto=0x7669, if1635087474, pkttype=PACKET_HOST, addr(0)={12652, }, 18) = 342

I'm using a setup (on our staging-system) with 2 root-servers and 1 fail-over-ip. As software we use corosync and pacemaker. Corosync is configured for multicast communication via port 5405. --> everything works fine.

Now I want to deploy this system on 2 root-server with failover-ip. Well, multicast communication won't work, because the root-servers are not directly connected ; they are connected with router and are located in different data-centers

Now I changed the corosync.conf according to udpu support (like the example). I'm using corosync v.1.4.1

corosync.conf:

compatibility: whitetank

totem {
        version: 2
        secauth: off
        interface {
                member {
                        memberaddr: A.A.A.A
                }
                member {
                        memberaddr: B.B.B.B
                }
                ringnumber: 0
                bindnetaddr:A.A.A.A
                mcastport: 5405
        }
transport: udpu
}

logging {
    fileline: off
        to_stderr: no
        to_logfile: yes
        to_syslog: yes
        logfile: /var/log/cluster/corosync.log
        debug: off
        timestamp: on
        logger_subsys {
                subsys: AMF
                debug: off
        }
}

service {
    name: pacemaker
    ver: 1
}

If I have a look at 'netstat -nlp' :

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:47150               0.0.0.0:*                   LISTEN      1224/rpc.statd      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1206/rpcbind        
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1314/sshd           
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1391/master         
tcp        0      0 :::111                      :::*                        LISTEN      1206/rpcbind        
tcp        0      0 :::22                       :::*                        LISTEN      1314/sshd           
tcp        0      0 :::45791                    :::*                        LISTEN      1224/rpc.statd      
udp        0      0 0.0.0.0:59806               0.0.0.0:*                               1769/corosync       
udp        0      0 0.0.0.0:39859               0.0.0.0:*                               1769/corosync       
udp        0      0 0.0.0.0:957                 0.0.0.0:*                               1206/rpcbind        
udp        0      0 0.0.0.0:56137               0.0.0.0:*                               1224/rpc.statd      
udp        0      0 0.0.0.0:976                 0.0.0.0:*                               1224/rpc.statd      
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1206/rpcbind        
udp        0      0 :::957                      :::*                                    1206/rpcbind        
udp        0      0 :::111                      :::*                                    1206/rpcbind        
udp        0      0 :::36209                    :::*                                    1224/rpc.statd

If I try a telnet on this port --> connection refused. Additionally I've disabeld SELinux, Firewall, etc. --> not working.

/var/log/cluster/corosync.log:

May 08 13:54:42 corosync [MAIN  ] Corosync Cluster Engine exiting with status 0 at main.c:1858.
May 09 12:22:18 corosync [MAIN  ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
May 09 12:22:18 corosync [MAIN  ] Corosync built-in features: nss dbus rdma snmp
May 09 12:22:18 corosync [MAIN  ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
May 09 12:22:18 corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
May 09 12:22:18 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
May 09 12:22:18 corosync [TOTEM ] bind token socket failed: Cannot assign requested address (99)
May 09 12:22:18 corosync [TOTEM ] The network interface [A.A.A.A] is now up.
Set r/w permissions for uid=0, gid=0 on /var/log/cluster/corosync.log
May 09 12:22:18 corosync [pcmk  ] info: process_ais_conf: Reading configure
May 09 12:22:18 corosync [pcmk  ] info: config_find_init: Local handle: 7739444317642555395 for logging
May 09 12:22:18 corosync [pcmk  ] info: config_find_next: Processing additional logging options...
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Found 'off' for option: debug
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Found 'yes' for option: to_logfile
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Found 'yes' for option: to_syslog
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
May 09 12:22:18 corosync [pcmk  ] info: config_find_init: Local handle: 5650605097994944516 for quorum
May 09 12:22:18 corosync [pcmk  ] info: config_find_next: No additional configuration supplied for: quorum
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: No default for option: provider
May 09 12:22:18 corosync [pcmk  ] info: config_find_init: Local handle: 2730409743423111173 for service
May 09 12:22:18 corosync [pcmk  ] info: config_find_next: Processing additional service options...
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Found '1' for option: ver
May 09 12:22:18 corosync [pcmk  ] info: process_ais_conf: Enabling MCP mode: Use the Pacemaker init script to complete Pacemaker startup
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_logd
May 09 12:22:18 corosync [pcmk  ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
May 09 12:22:18 corosync [pcmk  ] info: pcmk_startup: CRM: Initialized
May 09 12:22:18 corosync [pcmk  ] Logging: Initialized pcmk_startup
May 09 12:22:18 corosync [pcmk  ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
May 09 12:22:18 corosync [pcmk  ] info: pcmk_startup: Service: 10
May 09 12:22:18 corosync [pcmk  ] info: pcmk_startup: Local hostname: CentOS-60-64-minimal
May 09 12:22:18 corosync [pcmk  ] info: pcmk_update_nodeid: Local node id: 1632251470
May 09 12:22:18 corosync [pcmk  ] info: update_member: Creating entry for node 1632251470 born on 0
May 09 12:22:18 corosync [pcmk  ] info: update_member: 0x152edf0 Node 1632251470 now known as CentOS-60-64-minimal (was: (null))
May 09 12:22:18 corosync [pcmk  ] info: update_member: Node CentOS-60-64-minimal now has 1 quorum votes (was 0)
May 09 12:22:18 corosync [pcmk  ] info: update_member: Node 1632251470/CentOS-60-64-minimal is now: member
May 09 12:22:18 corosync [SERV  ] Service engine loaded: Pacemaker Cluster Manager 1.1.6
May 09 12:22:18 corosync [SERV  ] Service engine loaded: corosync extended virtual synchrony service
May 09 12:22:18 corosync [SERV  ] Service engine loaded: corosync configuration service
May 09 12:22:18 corosync [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
May 09 12:22:18 corosync [SERV  ] Service engine loaded: corosync cluster config database access v1.01
May 09 12:22:18 corosync [SERV  ] Service engine loaded: corosync profile loading service
May 09 12:22:18 corosync [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
May 09 12:22:18 corosync [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
May 09 12:22:27 corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
May 09 12:22:30 corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
May 09 12:22:32 corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
May 09 12:22:34 corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
May 09 12:22:36 corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.

If I try 'crm status' I get 'could not connect to cluster'

What is my fault? Can anyone help? A hint ?

I work as a Network Admin in a big company. Lately we noticed an issue with our network infrastructure. Basically our network backend lies on a Catalyst as main L3 backend switch, and few Cisco Nexus switches as edge L2 switches, connected to that Catalyst.

The issue appears as we try to sniff traffic on one of our hosts - we then (always) see unicast traffic between other hosts.

I'll try to be more elaborate: Assuming I'm on the host 10.0.0.1, with mac MAC, I run the command -

tcpdump -i eth0 ether host not MAC and host not 10.0.0.1 and not broadcast and not multicast

I will always see traffic between other hosts.

I read a Cisco article about Unicast Flooding, however - the "phenomenon" occurs not only when passing between VLANs in our network, but also on the on the very same VLAN. Is it possible that it happens when passing between switches in the same VLAN (our VLANs span on many switches)? All switches are connected by a trunk to the Catalyst...

Any ideas?

Thanks.

Edit:

It seems that we found the source of our problems.

Basically, each time one of the switches gets a frame with a MAC address it doesn't recognize - it floods it to all ports. This is normal - and the way things should go. However, in our current settings, a MAC entry in the switch should "live" for 30 minutes. If a MAC was not seen for 30 minutes, it will be deleted from the switch until seen again. If a packet is sent to that MAC and it's not in the table - all ports will be flooded in order to find the destination MAC port (we expect to get an answer from one of the ports).

We found one of the destination MACs and looked for it in the switch MAC table. The table didn't contain the MAC while the network was flooded. We tried ARPing the address related to that MAC - and the flood stopped (as the MAC re-appeared in the MAC table).
However, after a few seconds, the MAC disappeared from the MAC table again and the flood started again.

It seems that the flood issue derives from an issue with the MAC tables on our switches. It seems as if they "forget" MAC addresses quickly than they should (MACs should stay for 30 minutes) and flood all packets with that MAC.

We had a 2 node NLB cluster running IIS websites on virtual machines. Both nodes were online, the balancer functioned exactly as expected, (if traffic is 50/50 balanced, and you stop or drainstop a node, all traffic routes seamlessles to the other node.)

But when I rebooted a node, even if I stopped it prior to reboot, the OTHER node which should be receiving production traffic during the reboot stopped accepting requests.

To my knowledge this was NOT how NLB is supposed to work. If I power down a node, the other nodes in the NLB cluster shouldn't care, and should continue to accept traffic according to their port rules while the offline node reboots.

None of my port rules employed affinity, so I knew that wasn't the issue.

So after agonizing over it a bit, I stumbled on the answer (see my posted answer)