Home / server / Questions / 1044516
Accepted
milosgajdos
Asked: 2020-12-02 03:00:20 +0800 CST

How to use ss in specific network namespace

  • 772

Whilst investigating some TCP network issues in containers I tried to use ss to peek into container network TCP stack.

We are running Amazon Linux in AWS:

# uname -a
Linux 4.14.173-137.229.amzn2.x86_64 #1 SMP Wed Apr 1 18:06:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

ss has the following cli switch for that:

       -N NSNAME, --net=NSNAME
              Switch to the specified network namespace name.

lsns gives me the following output:

# lsns | grep net
4026531993 net       225     1 root      /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532284 net         2 26244 root      /pause

This is pause container created for each Kubernetes pod -- it's the container that creates the network namespace.

Trying to peek into the pod network namespace by running ss:

# ss -tp -N 4026532284
Cannot open network namespace "4026532284": No such file or directory

What's interesting is ip netns list does not return any network namespaces:

# ip netns list
#

Is there any way how one can look into K8s pod network namespaces from the root network namespace i.e. from the netns 1?

# ss --version
ss utility, iproute2-ss180129

# lsns --version
lsns from util-linux 2.30.2

# rpm -qi iproute
Name        : iproute
Version     : 4.15.0
Release     : 1.amzn2.0.4
Architecture: x86_64
Install Date: Sat 07 Mar 2020 03:42:24 AM UTC
Group       : Applications/System
Size        : 1321292
License     : GPLv2+ and Public Domain
Signature   : RSA/SHA256, Fri 21 Feb 2020 09:00:29 PM UTC, Key ID 11cf1f95c87f5b1a
Source RPM  : iproute-4.15.0-1.amzn2.0.4.src.rpm
Build Date  : Fri 21 Feb 2020 07:56:50 PM UTC
Build Host  : build.amazon.com
Relocations : (not relocatable)
Packager    : Amazon Linux
Vendor      : Amazon Linux
URL         : http://kernel.org/pub/linux/utils/net/iproute2/
Summary     : Advanced IP routing and network device configuration tools

Update: Tue Dec 1 11:35:39 UTC 2020

After some struggle, I finally decided to strace this.

It turns out ss is an awesome tool, but when it comes to using it with containers it leaves a bit to be desired, but I feel there is more than one "culprit" involved.

ss does not bother to look up the actual PID of the process that creates the network namespaces, but rather goes directly to check /var/run/netns:

openat(AT_FDCWD, "/var/run/netns/4026532284", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "Cannot open network namespace \"4"..., 70Cannot open network namespace "4026532284": No such file or directory
) = 70

Now, I suspect this is due to how iproute package creates network namespaces i.e. given the ss is shipped with iproute package the assumption ip makes about network namespaces is: "hey all network ns should be found in /var/run/netns directory, because, like, why not and also this will make lives of iproute devs easy, or whatever.

Turns out that's a false assumption made on ss/iproute side or the lack of "agreement" on modern containers tools and iproute interoperability, but it kind of explains the empty output from

ip netns list

So the way ip creates network namespaces (so they can be inspected by ss) obviously does not match how kubernetes and the likes create them, making iproute package utilities borderline useless in the grand scheme of things.

networking linux-networking containers network-namespace

3 Answers

  1. Best Answer

    A more generic way is to use nsenter(1).

    nsenter -t ${PID_FOO} -muni ss -tpi

    A go to approach is using something like the following, when needing to run adhoc stuff, that do not necessarily have support for unshare(2)/setns(2) built-in.

    docker run -it --rm --security-opt=seccomp:unconfined \
                    --security-opt=apparmor:unconfined \
                    --privileged --pid=host --userns=host \
debian:jessie@sha256:51cd80bb935b76fbbf49640750736abc63ab7084d5331e198326b20063e7f13c \
nsenter -t ${PID_FOO} -m -u -n -i -F ss -tpi
    • 2

  2. If you want to use ss to peek into a particular container namespace here is how you do it:

    1. Find out the PID of the container process ps aux or ps -ef should give you the answer

    2. Create the following symlink

    ln -s /proc/PID/ns/net /var/run/netns/mycontainer
    1. Profit
    ss -tpi -N mycontainer
    • 1

  3. If you have a recent version of lsns, you can use options -n -o NSFS to convert a namespace inode into the ID used by the network subsystem.

    For example, let's say you have net NS 4026536974. You can run:

    sh-4.4# lsns --version
lsns from util-linux 2.32.1
sh-4.4# lsns -n -o NSFS 4026536974 | sort -u                                                                                                                                                                                                 
/run/netns/d0912eba-0fae-425c-94ba-cf270aa23c93
sh-4.4# basename /run/netns/d0912eba-0fae-425c-94ba-cf270aa23c93                                                                                                                                                                             
d0912eba-0fae-425c-94ba-cf270aa23c93
sh-4.4# ss -nltp -N d0912eba-0fae-425c-94ba-cf270aa23c93 | head -2                                                                                                                                                                           
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port                                                                                    
LISTEN    0         128                0.0.0.0:5000             0.0.0.0:*        users:(("nginx",pid=874035,fd=5),("nginx",pid=874028,fd=5))                    
sh-4.4#

    Or all in one:

    sh-4.4# lsns -n -o NSFS 4026536974 | sort -u | xargs -rn1 basename | xargs -rn1 ss -nltp -N | head -2                                                                                                                                        
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port                                                                                    
LISTEN    0         128                0.0.0.0:5000             0.0.0.0:*        users:(("nginx",pid=874035,fd=5),("nginx",pid=874028,fd=5))                    
sh-4.4#
    • 0