Home / server / Questions / 1127792
Accepted
Maciek Leks
Asked: 2023-04-03 18:50:40 +0800 CST

microk8s communication - vxlan.calico, bridge and veth

  • 772

I've configured my microk8s instance (one node). Works well. I've started digging in some linux network internals and I was dumbfounded looking at this:

$ ip -c -br link

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
enp0s3           UP             08:00:27:ad:36:a3 <BROADCAST,MULTICAST,UP,LOWER_UP> 
enp0s8           UP             08:00:27:86:35:40 <BROADCAST,MULTICAST,UP,LOWER_UP> 
enp0s9           UP             08:00:27:25:c8:6d <BROADCAST,MULTICAST,UP,LOWER_UP> 
br-e6dc1065d537  DOWN           02:42:92:65:b3:c3 <NO-CARRIER,BROADCAST,MULTICAST,UP> 
vxlan.calico     UNKNOWN        66:57:ed:b9:9c:1c <BROADCAST,MULTICAST,UP,LOWER_UP> 
cali9e5a2199a75@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
caliec07808c9ad@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
calid39e8c9a8ec@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP> 
cali6a901a5bd4a@if3 UP             ee:ee:ee:ee:ee:ee <BROADCAST,MULTICAST,UP,LOWER_UP>

$ brctl show br-e6dc1065d537

bridge name bridge id       STP enabled interfaces
br-e6dc1065d537     8000.02429265b3c3   no

As you can see linux bridge is DOWN, and it does not have anything connected to it, e.g. calico veths.

So I've started sniffing traffic on the vxlan.calico vxlan when I curl some workloads between each other. Nothing there (on that device). No traffic found. As far as I know and I've tested to communicate veths you need a bridge. To run vxlan over it you need vxlan be connected to the bridge,...But as you see nothing is here.

The only communication I see is on the veths, but how can it be, that two unpaired veths talks to each other?

microk8s

1 Answers

  1. Best Answer

    OK, the truth is quite simple. For pod-to-pod communications with a single node case there is no need to use bridging, so the bridge is DOWN (I suppose), while the whole communications is based on routing routes. ip -br -c route shows the true and the way calico manages routes dynamically.

    $ ip -br -c route

    default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100 
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 
10.0.2.3 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100 
blackhole 10.1.238.128/26 proto 80 
10.1.238.138 dev cali6a901a5bd4a scope link 
10.1.238.142 dev caliec07808c9ad scope link 
10.1.238.146 dev cali9e5a2199a75 scope link 
10.1.238.149 dev cali00db5abf42d scope link 
192.168.57.0/24 dev enp0s9 proto kernel scope link src 192.168.57.4 metric 100 
192.168.59.0/24 dev enp0s8 proto kernel scope link src 192.168.59.118 metric 100

    One thing puzzled me. If we look at the interfaces all calico veths in the root namespace have got the same MACs: ee:ee:ee:ee:ee:ee, while

    $ ips neigh

    shows paired veth MACs in container namespaces

    10.1.238.149                            cali00db5abf42d  2e:b1:c3:59:e4:09
10.1.238.138                            cali6a901a5bd4a  a6:b8:a9:c3:e7:77
10.1.238.146                            cali9e5a2199a75  6e:c4:7d:a1:3b:d2
10.1.238.142                            caliec07808c9ad  32:dc:93:ed:eb:51

    So it means that devices in the root ns don't need "correct" MACs since we've got routings and IP resolution table.

    • 0