icelava's questions

We have certain Ubuntu (18.04) servers operating .NET Core apps on Kestrel. Recently the apps and OSes have been hanging, requiring entire VM restart.

What we found was the app had way too many file descriptors for network sockets. i.e. /proc/

Correspondingly, a query with lsof would reveal an ever-growing pool of "protocol:TCP" of type sock the dotnet process leaks.

COMMAND  PID     USER   FD      TYPE             DEVICE SIZE/OFF   NODE NAME
dotnet  6905 www-data  675u     sock                0,9      0t0  96816 protocol: TCP
dotnet  6905 www-data  676u     sock                0,9      0t0  96863 protocol: TCP
dotnet  6905 www-data  677u     sock                0,9      0t0  96910 protocol: TCP
dotnet  6905 www-data  678u     sock                0,9      0t0  96959 protocol: TCP
dotnet  6905 www-data  679u     sock                0,9      0t0  97006 protocol: TCP
dotnet  6905 www-data  680u     sock                0,9      0t0  97053 protocol: TCP
dotnet  6905 www-data  681u     sock                0,9      0t0  97101 protocol: TCP

From another thread with similar reported problem, https://serverfault.com/questions/153983/sockets-found-by-lsof-but-not-by-netstat we understand that is due to some code (likely some library) in the .NET app that is opening sockets without binding IP address or port to it.

However, what we particularly like to know in this case is, what are DEVICE 0 and 9 in this context? I have seen other reports list 0,5 or 0,7 but nobody explains what those numbers represent which type of device.

After seventeen years since last touching Linux (Red Hat), I've come back to work on Ubuntu 18.04 servers. I see plenty have changed, including the foundational init system systemd, which appeared to cause some controversy and division among the community.

I've learnt that network interface configuration is now handled by a program called netplan, of which I have been able to edit the /etc/netplan/50-cloud-init.yaml file to persist interface settings, like for my simple virtual machine below:

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled} network:
    ethernets:
        eth0:
            addresses:
            - 192.168.0.21/24
            gateway4: 192.168.0.1
            nameservers:
                addresses:
                - 165.21.100.88
                - 165.21.83.88
        eth1:
             addresses:
             - 192.168.7.11/24
    version: 2

The comments of that file however confuses me; it claims this setup is take from some other "datasource" - where? But when I read https://netplan.io/design it indicates that is the authoritative source for network configuration.

That of which I understand will then be handed to "network renderers" which I guess here means rendering netplan's config format into another text format for systemd-networkd(8) (what's the difference from systemd.network(5)? Unsure what the 8 vs 5 means either.) or NetworkManager to handle the actual runtime device/network configuration, rather than the more traditional meaning of distributed image/3D rendering that Google suggests.

Anyway, in the context of the actual production servers, they could be configured with one to three more secondary ethernet interfaces each with their own subnet. Simple example:

network:
    ethernets:
        eth0:
            addresses:
            - 10.0.1.100/24
            gateway4: 10.0.1.1
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
        eth1:
            addresses:
            - 10.0.2.100/24
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
    version: 2

From our control network, say 10.0.0.0/24, we can route to both 10.0.1.0/24 and 10.0.2.0/24. However the server above can really only be contactable (be it HTTP or ICMP ping) at eth0 10.0.1.100. Traffic to 10.0.2.100 never connect.

https://brainscraps.fandom.com/wiki/Setup_Gateway_Routing_On_Multiple_Network_Interfaces explains it's because Linux on its own does not know how to route replies if its recipient destination is 10.0.2.100 (eth1). It (each extra interface) needs its own routing table to respond via its proper gateway 10.0.2.1 just like eth0's 10.0.1.1. However, that page deals with the traditional style of network configuration which netplan has superceded. Furthermore I read that netplan does not support rt2 tables? (iproute2?)

What should the kosher approach be for netplan? Could it be as simple as supplying its own gateway4?

        eth1:
            addresses:
            - 10.0.2.100/24
            gateway4: 10.0.2.1

Or is it a lot more involved configuration similar to the iproute2 effort? I'm not sure how to test on the actual server (cannot test this on my own VM since I don't have the same multi-network hardware and environment in my laptop.) for fear of losing connection to it (SSH) after mis-configuration.

UPDATE 17 SEP: Today we discussed and went ahead with directly modifying the server's 50-cloud-init.yaml configuration to include the above extra gateway for eth1, but the server still can't route out responses from that interface.

UPDATE 17 SEP EVENING: Tried using explicit route instead of gateway4

        eth0:
            addresses:
            - 10.0.2.100/24
#            gateway4: 10.0.2.1
            routes:
            - to: 0.0.0.0/0
              via: 10.0.2.1
              metric: 100

but still can't route. From tcpdump I can see ICMP ping coming in (from my laptop) but no outgoing reply.