Home / server / Questions / 1019487
In Process
Steve Fowlkes
Asked: 2020-06-01 19:28:24 +0800 CST

Route Table Multiple Subnet Entries?

  • 772

So just as a bit of context, this is for a VM running that does call processing. So I was looking at the route table on the VM, and noticed that it has this:

  12.12.12.64/28 dev bond15 proto kernel scope link src 12.12.12.12
  12.12.12.64/28 dev bond19 proto kernel scope link src 12.12.12.12
  12.12.12.64/28 dev bond11 proto kernel scope link src 12.12.12.12
  12.12.12.64/28 dev bond8 proto kernel scope link src 12.12.12.12

So my question is, does this make sense? I didn't add them to the routing table on my own these were set when the IP's were configured. But my question is, how does this work then? If a packet destined to anything on 12.12.12.64/28 comes in it's always going to go to the bond15 interface correct? Wouldn't this completely negate the bond19 and bond11 interfaces?

Now that being said, I also did an ifconfig of the VM, and noticed that all of these BOND interfaces, have the same IP:

  bond8: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1472
          inet 12.12.12.12  netmask 255.255.255.240  destination 12.12.12.12
 bond11: flags=209<UP,POINTOPOINT,RUNNING,NOARP>         mtu 1472
          inet 12.12.12.12  netmask 255.255.255.240  destination 12.12.12.12
        bond15: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1472
    inet 12.12.12.12  netmask 255.255.255.240    destination 12.12.12.12
  bond19: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1472
    inet 12.12.12.12  netmask 255.255.255.240  destination 12.12.12.12

So given these two pieces of information, I'm curious as to how to pull this all together. Does this mean that all of these BOND's are just using the same interface? If so why does there need to be multiple lines in the route table? and why not just have one single entry in the ifconfig that consolidates all 4 Bonds into one? So there's just one Bond0 for 12.12.12.12 and on ipconfig entry for Bond0 for 12.12.12.12 ?

The way it reads to me is that according to the route table, it any packet destined for this subnet will only go to Bond15 because it appears first in the route list?

Thanks all..

routing subnet route ifconfig

1 Answers

  1. First you should know that for any advanced network settings, ifconfig and route should be entirely avoided, and ip link, ip address, ip route (as well as ip rule for policy routing, bridge in addition to ip link to replace brctl) used instead. Some features are not available with older tools because newer features are implemented only in the newer kernel API ((rt)netlink instead of ioctl).

    These routes are tagged proto kernel: that means they were automatically added by the kernel when adding addresses to the interfaces, probably with something like:

    ip address add 12.12.12.12 peer 12.12.12.64/28 dev bond8

    As is alone it doesn't make much sense: only the first listed route among equals would be used: bond15, there has to be other elements for a correct routing.

    I can think of at least two ways this is done:

    • additional routing tables

      This would probably have to be working along with conntrack zones (to avoid mixing two identical flows except their interface), connmarks and marks (using iptables or nftables) to mark packets to know the route they came from and allowing the kernel to reply back there. The ip rule command returns this on a standard system:

      0:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default

      If you have additional entries, that's probably the method chosen, and it could become quite complex if additional settings are done using iptables or nftables. For each additional entry with lookup XXXX, you can display the additional routing table with:

      ip route show table XXXX

      which will probably have only one bondXX device referenced inside it (maybe along other interfaces).

    • binding directly to an interface

      Using SO_BINDTODEVICE:

      SO_BINDTODEVICE

      Bind this socket to a particular device like “eth0”, as specified in the passed interface name. [...] If a socket is bound to an interface, only packets received from that particular interface are processed by the socket. Note that this works only for some socket types, particularly AF_INET sockets.

      This would usually require to leave Reverse Path Forwarding filter off for example with:

      sysctl -w net.ipv4.conf.all.rp_filter=0

sysctl -w net.ipv4.conf.bond8.rp_filter=0
sysctl -w net.ipv4.conf.bond11.rp_filter=0
sysctl -w net.ipv4.conf.bond15.rp_filter=0
sysctl -w net.ipv4.conf.bond19.rp_filter=0

      actually probably simply with this before creation of the interfaces:

      sysctl -w net.ipv4.conf.default.rp_filter=0

      I could get this working using socat and its so-bindtodevice= option, but only as client (connecting), not as server (listening) so I don't know if it's for all cases, or if that was a limitation with socat or something I missed. Something like:

      socat tcp4:12.12.12.65:5555,so-bindtodevice=bond11 -

      to communicate with the service on port 5555/tcp on the system owning 12.12.12.65 on the bond11 side (rather than the one on the bond15 side).

    Also despite their name, these interfaces don't appear to be of type bond, but probably some kind of tunnel. Using ip -details link show would display their actual type (at the start of the 3rd line for each interface).

    • 0