What I am trying to do?
I'm trying to acquire 3 public IP addresses via DHCP on a single physical ISP upstream cable.
What goes wrong?
Renewals go kind of wrong. From interfaces, virtual0 works just fine when it tries to renew via unicast connection to DHCP server. virtual1 and virtual2 interfaces fail with unicast. They drop into fallback mode to use broadcasts and succeed with them, but spam the logs full of stuff which cannot be good. I'm running Debian 8 and isc-dhcp-client v. 4.3.1 .
I set up the interfaces on boot like this:
IF_VIRTUAL_BASE=eth0 IF_PUB0=virtual0 IF_PUB1=virtual1 IF_PUB2=virtual2 IF_LAN=eth2 ifconfig "$IF_VIRTUAL_BASE" up ip link add link "$IF_VIRTUAL_BASE" address 00:90:0b:ff:10:5b "$IF_PUB0" type macvlan ip link set "$IF_PUB0" up ip link add link "$IF_VIRTUAL_BASE" address 00:90:0b:ff:11:5b "$IF_PUB1" type macvlan ip link set "$IF_PUB1" up ip link add link "$IF_VIRTUAL_BASE" address 00:90:0b:ff:12:5b "$IF_PUB2" type macvlan ip link set "$IF_PUB2" up # Enable forwarding echo 1 > /proc/sys/net/ipv4/ip_forward
I also do unset new_routers when completing dhcprequest for virtual1 and virtual2.
And guess what, this monstrosity kind of works:
Jul 11 20:45:43 gw dhclient: DHCPREQUEST on virtual0 to 255.255.255.255 port 67 Jul 11 20:45:43 gw dhclient: DHCPREQUEST on virtual1 to 255.255.255.255 port 67 Jul 11 20:45:43 gw ifup[582]: DHCPREQUEST on virtual0 to 255.255.255.255 port 67 Jul 11 20:45:43 gw ifup[592]: DHCPREQUEST on virtual1 to 255.255.255.255 port 67 Jul 11 20:45:43 gw dhclient: DHCPREQUEST on virtual2 to 255.255.255.255 port 67 Jul 11 20:45:43 gw ifup[634]: DHCPREQUEST on virtual2 to 255.255.255.255 port 67 Jul 11 20:45:46 gw dhclient: DHCPREQUEST on virtual1 to 255.255.255.255 port 67 Jul 11 20:45:46 gw ifup[592]: DHCPREQUEST on virtual1 to 255.255.255.255 port 67 Jul 11 20:45:46 gw dhclient: DHCPACK from 88.113.75.2 Jul 11 20:45:46 gw ifup[592]: DHCPACK from 88.113.75.2 Jul 11 20:45:46 gw logger: virtual1 (REBOOT): IP: -> 88.113.75.59; GW: -> 88.113.75.1 Jul 11 20:45:47 gw dhclient: DHCPREQUEST on virtual0 to 255.255.255.255 port 67 Jul 11 20:45:47 gw ifup[582]: DHCPREQUEST on virtual0 to 255.255.255.255 port 67 Jul 11 20:45:47 gw dhclient: DHCPACK from 88.113.75.2 Jul 11 20:45:47 gw ifup[582]: DHCPACK from 88.113.75.2 Jul 11 20:45:47 gw logger: virtual0 (REBOOT): IP: -> 88.113.75.65; GW: -> 88.113.75.1 Jul 11 20:45:50 gw dhclient: DHCPREQUEST on virtual2 to 255.255.255.255 port 67 Jul 11 20:45:50 gw ifup[634]: DHCPREQUEST on virtual2 to 255.255.255.255 port 67 Jul 11 20:45:50 gw dhclient: DHCPACK from 88.113.75.2 Jul 11 20:45:50 gw ifup[634]: DHCPACK from 88.113.75.2 Jul 11 20:45:50 gw logger: virtual2 (REBOOT): IP: -> 88.113.75.61; GW: -> 88.113.75.1
So they worked, right? And they are broadcast requests. Ok, fast forward an hour or so we gets looots of this:
Jul 11 21:45:09 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:45:11 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:45:22 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:45:25 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:45:32 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:45:43 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:45:46 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:45:56 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:46:05 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:46:11 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:46:21 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:46:30 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:46:33 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:46:42 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:46:47 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:47:00 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 21:47:01 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 21:47:09 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67
The virtual1 and virtual2 interfaces desperately try to renew their IPs via unicast connection to previous DHCP server 195.74.6.55 . They fail for the unicast. But something funny happens next, in the end they switch to broadcast as fallback and succeed!
Jul 11 22:30:41 gw dhclient: DHCPREQUEST on virtual1 to 195.74.6.55 port 67 Jul 11 22:30:47 gw dhclient: DHCPREQUEST on virtual2 to 195.74.6.55 port 67 Jul 11 22:30:52 gw dhclient: DHCPREQUEST on virtual1 to 255.255.255.255 port 67 Jul 11 22:30:52 gw dhclient: DHCPACK from 88.113.75.2 Jul 11 22:30:52 gw logger: virtual1 (RENEW): IP: 88.113.75.59 -> 88.113.75.59; GW: 88.113.75.1 -> 88.113.75.1 Jul 11 22:30:58 gw dhclient: DHCPREQUEST on virtual2 to 255.255.255.255 port 67 Jul 11 22:30:58 gw dhclient: DHCPACK from 88.113.75.2 Jul 11 22:30:59 gw logger: virtual2 (RENEW): IP: 88.113.75.61 -> 88.113.75.61; GW: 88.113.75.1 -> 88.113.75.1
Observe the virtual0 interface:
Jul 11 22:38:17 gw dhclient: DHCPREQUEST on virtual0 to 195.74.6.55 port 67 Jul 11 22:38:18 gw dhclient: DHCPACK from 195.74.6.55 Jul 11 22:38:18 gw logger: virtual0 (RENEW): IP: 88.113.75.65 -> 88.113.75.65; GW: 88.113.75.1 -> 88.113.75.1
Conclusion is that for virtual0 interface, unicast (dhclient) DHCP requests work and for virtual1 and virtual2 only broadcast DHCP renewals work. So, it must be a routing problem, right? Here is what ip route shows after a typical boot:
root@gw:~# ip route default via 88.113.75.1 dev virtual0 88.113.75.0/24 dev virtual0 proto kernel scope link src 88.113.75.65 88.113.75.0/24 dev virtual1 proto kernel scope link src 88.113.75.59 88.113.75.0/24 dev virtual2 proto kernel scope link src 88.113.75.61 172.16.8.0/28 via 172.16.8.2 dev tun0 # For openvpn 172.16.8.0/24 dev eth2 proto kernel scope link src 172.16.8.254 # For LAN 172.16.8.2 dev tun0 proto kernel scope link src 172.16.8.1 #For openvpn
How do I make the dhclients on macvlan-based interfaces virtual1 and virtual2 route their unicast renewal requests correctly and receive the responses appropriately?
I have done tons of Google searches, tried to flip-flop many settings, including firewall policies, policy-based routings, eth0 promisc mode and sysctl variables, stripping down the generic networking setup etc. I'm currently doing strace on dhclient. One combination of these settings must work. If more information is needed, I'm more than happy to provide.
EDIT1: dhclient strace became ready.
This happens in the very beginning:
bind(5, {sa_family=AF_PACKET, proto=0x7669, if1635087474, pkttype=PACKET_HOST, addr(0)={12652, }, 16) = 0
bind(6, {sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
sendto(5, "......", 342, 0, {sa_family=AF_PACKET, proto=0x7669, if1635087474, pkttype=PACKET_HOST, addr(0)={12652, }, 18) = 342
The "sendto(5" was, to my understanding, the first successful broadcast-based dhclient send.
Then, later the first failing unicast send:
sendto(6, "......", 300, 0, {sa_family=AF_INET, sin_port=htons(67), sin_addr=inet_addr("195.74.6.55")}, 16) = 300
And the next successful broadcast send:
sendto(5, "......", 342, 0, {sa_family=AF_PACKET, proto=0x7669, if1635087474, pkttype=PACKET_HOST, addr(0)={12652, }, 18) = 342