Whilst investigating some TCP network issues in containers I tried to use ss
to peek into container network TCP stack.
We are running Amazon Linux in AWS:
# uname -a
Linux 4.14.173-137.229.amzn2.x86_64 #1 SMP Wed Apr 1 18:06:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
ss
has the following cli switch for that:
-N NSNAME, --net=NSNAME
Switch to the specified network namespace name.
lsns
gives me the following output:
# lsns | grep net
4026531993 net 225 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 21
4026532284 net 2 26244 root /pause
This is pause
container created for each Kubernetes
pod -- it's the container that creates the network namespace.
Trying to peek into the pod network namespace by running ss
:
# ss -tp -N 4026532284
Cannot open network namespace "4026532284": No such file or directory
What's interesting is ip netns list
does not return any network namespaces:
# ip netns list
#
Is there any way how one can look into K8s pod network namespaces from the root network namespace i.e. from the netns 1?
# ss --version
ss utility, iproute2-ss180129
# lsns --version
lsns from util-linux 2.30.2
# rpm -qi iproute
Name : iproute
Version : 4.15.0
Release : 1.amzn2.0.4
Architecture: x86_64
Install Date: Sat 07 Mar 2020 03:42:24 AM UTC
Group : Applications/System
Size : 1321292
License : GPLv2+ and Public Domain
Signature : RSA/SHA256, Fri 21 Feb 2020 09:00:29 PM UTC, Key ID 11cf1f95c87f5b1a
Source RPM : iproute-4.15.0-1.amzn2.0.4.src.rpm
Build Date : Fri 21 Feb 2020 07:56:50 PM UTC
Build Host : build.amazon.com
Relocations : (not relocatable)
Packager : Amazon Linux
Vendor : Amazon Linux
URL : http://kernel.org/pub/linux/utils/net/iproute2/
Summary : Advanced IP routing and network device configuration tools
Update: Tue Dec 1 11:35:39 UTC 2020
After some struggle, I finally decided to strace
this.
It turns out ss
is an awesome tool, but when it comes to using it with containers it leaves a bit to be desired, but I feel there is more than one "culprit" involved.
ss
does not bother to look up the actual PID of the process that creates the network namespaces, but rather goes directly to check /var/run/netns
:
openat(AT_FDCWD, "/var/run/netns/4026532284", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "Cannot open network namespace \"4"..., 70Cannot open network namespace "4026532284": No such file or directory
) = 70
Now, I suspect this is due to how iproute
package creates network namespaces
i.e. given the ss
is shipped with iproute
package the assumption ip
makes about network namespaces is: "hey all network ns should be found in /var/run/netns
directory, because, like, why not and also this will make lives of iproute
devs easy, or whatever.
Turns out that's a false assumption made on ss
/iproute
side or the lack of "agreement" on modern containers tools and iproute
interoperability, but it kind of explains the empty output from
ip netns list
So the way ip
creates network namespaces (so they can be inspected by ss
) obviously does not match how kubernetes and the likes create them, making iproute
package utilities borderline useless in the grand scheme of things.