Home / ubuntu / Questions / 1174504
Accepted
icelava
Asked: 2019-09-17 03:02:34 +0800 CST

18.04 server unreachable on secondary network interfaces

  • 772

After seventeen years since last touching Linux (Red Hat), I've come back to work on Ubuntu 18.04 servers. I see plenty have changed, including the foundational init system systemd, which appeared to cause some controversy and division among the community.

I've learnt that network interface configuration is now handled by a program called netplan, of which I have been able to edit the /etc/netplan/50-cloud-init.yaml file to persist interface settings, like for my simple virtual machine below:

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled} network:
    ethernets:
        eth0:
            addresses:
            - 192.168.0.21/24
            gateway4: 192.168.0.1
            nameservers:
                addresses:
                - 165.21.100.88
                - 165.21.83.88
        eth1:
             addresses:
             - 192.168.7.11/24
    version: 2

The comments of that file however confuses me; it claims this setup is take from some other "datasource" - where? But when I read https://netplan.io/design it indicates that is the authoritative source for network configuration.

That of which I understand will then be handed to "network renderers" which I guess here means rendering netplan's config format into another text format for systemd-networkd(8) (what's the difference from systemd.network(5)? Unsure what the 8 vs 5 means either.) or NetworkManager to handle the actual runtime device/network configuration, rather than the more traditional meaning of distributed image/3D rendering that Google suggests.

Anyway, in the context of the actual production servers, they could be configured with one to three more secondary ethernet interfaces each with their own subnet. Simple example:

network:
    ethernets:
        eth0:
            addresses:
            - 10.0.1.100/24
            gateway4: 10.0.1.1
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
        eth1:
            addresses:
            - 10.0.2.100/24
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
    version: 2

From our control network, say 10.0.0.0/24, we can route to both 10.0.1.0/24 and 10.0.2.0/24. However the server above can really only be contactable (be it HTTP or ICMP ping) at eth0 10.0.1.100. Traffic to 10.0.2.100 never connect.

https://brainscraps.fandom.com/wiki/Setup_Gateway_Routing_On_Multiple_Network_Interfaces explains it's because Linux on its own does not know how to route replies if its recipient destination is 10.0.2.100 (eth1). It (each extra interface) needs its own routing table to respond via its proper gateway 10.0.2.1 just like eth0's 10.0.1.1. However, that page deals with the traditional style of network configuration which netplan has superceded. Furthermore I read that netplan does not support rt2 tables? (iproute2?)

What should the kosher approach be for netplan? Could it be as simple as supplying its own gateway4?

        eth1:
            addresses:
            - 10.0.2.100/24
            gateway4: 10.0.2.1

Or is it a lot more involved configuration similar to the iproute2 effort? I'm not sure how to test on the actual server (cannot test this on my own VM since I don't have the same multi-network hardware and environment in my laptop.) for fear of losing connection to it (SSH) after mis-configuration.

UPDATE 17 SEP: Today we discussed and went ahead with directly modifying the server's 50-cloud-init.yaml configuration to include the above extra gateway for eth1, but the server still can't route out responses from that interface.

UPDATE 17 SEP EVENING: Tried using explicit route instead of gateway4

        eth0:
            addresses:
            - 10.0.2.100/24
#            gateway4: 10.0.2.1
            routes:
            - to: 0.0.0.0/0
              via: 10.0.2.1
              metric: 100

but still can't route. From tcpdump I can see ICMP ping coming in (from my laptop) but no outgoing reply.

netplan systemd-networkd

1 Answers

  1. Best Answer

    Ok looks like as per original old article, these extra routes truly need to be defined into separate routing tables, coupled with explicit routing-policy directives for "force" the OS to route traffic to/from the interfaces IP addresses with those extra tables.

    network:
    ethernets:
        eth0:
            addresses:
            - 10.0.1.100/24
            gateway4: 10.0.1.1
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
        eth1:
            addresses:
            - 10.0.2.100/24
# Simply defining gateway for secondary interface is not good enough.
#            gateway4: 10.0.2.1

# Needs a separate route table with a to-ANY route that specifies its default gateway.
# Metric doesn't matter since it's not supposed to compete with primary interface.
            routes:
            - to: 0.0.0.0/0
              via: 10.0.2.1
              metric: 100
              table: 100

# Declare the cases when the other route table is to be used.
# Whenever the dest = interface IP address
# Whenever the src = interface IP address
            routing-policy:
            - to: 10.0.2.100/24
              table: 100
            - from: 10.0.2.100/24
              table: 100
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
    version: 2

    The second network interface and IP address is now contactable.

    • 1