Because I like to have things separate (security concern) I’m currently experimenting to see if I can have completely separate networks in Kubernetes for the “hosts” network and the “external” network where I want MetalLB to manage the traffic.
As an experimentation setup I now have two separate vlans and a VMware virtual machine which simply sees two normal network interfaces.
On that VM I have installed CentOS 7, Kubernetes 1.17, Calico 3.13.1 and MetalLB 0.8.2
I expect that I do not need to give the node an IP address on the “external” interface because in the MetalLB manual I read
Layer 2 mode does not require the IPs to be bound to the network interfaces of your worker nodes. It works by responding to ARP requests on your local network directly, to give the machine’s MAC address to clients.
This is nice because that way the node is effectively unreachable from the "external" network.
For my testing I have configured metallb to manage a range of IP on both networks (just for testing)
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 10.123.123.100-10.123.123.199
- 10.123.45.100-10.123.45.199
On the VM I give the host interface an ip address (DHCP) and that all works fine. I do not give the “external” interface an IP/routing/… yet I have made sure it is “up”.
So I see this (ens192 = hosts, ens224 = external)
[root@node1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:ca:06:13 brd ff:ff:ff:ff:ff:ff
inet 10.123.123.6/24 brd 10.123.123.255 scope global dynamic ens192
valid_lft 75786sec preferred_lft 75786sec
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:ca:06:1d brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:dc:25:87:ec brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
(and a lot of cali* interfaces)
[root@node1 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.123.123.1 0.0.0.0 UG 0 0 0 ens192
10.123.123.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.135.192 0.0.0.0 255.255.255.192 U 0 0 0 *
192.168.135.242 0.0.0.0 255.255.255.255 UH 0 0 0 calia47abf08c66
…
For testing I have a simple web app and I’ve attached two K8s services to it: One with a loadBalancerIP on the “host” network and one on the “external” network.
[root@node1 ~]# kubectl -n yauaa get all
NAME READY STATUS RESTARTS AGE
pod/yauaa-7d7bffd9fc-tdhg9 1/1 Running 5 2d6h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/yauaa LoadBalancer 10.104.33.179 10.123.123.100 80:31908/TCP 2d3h
service/yauaa-pub LoadBalancer 10.105.4.161 10.123.45.100 80:30695/TCP 2d7h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/yauaa 1/1 1 1 2d7h
NAME DESIRED CURRENT READY AGE
replicaset.apps/yauaa-7d7bffd9fc 1 1 1 2d6h
Running locally on the k8s node itself I can do a curl to either IP and both respond immediately. So the binding provided by metallb to the services seems to work.
Now on my firewall (a linux device) I can do a curl to either IP, the “host” one responds, the “external” one does not. I do see that the correct IP-MAC mapping is present in the arp table of this firewall. Yet I do not get a response.
# arp -a | fgrep 10.123
node1.k8s.basjes.nl (10.123.123.6) at 00:0c:29:ca:06:13 [ether] on eth1.2222
? (10.123.123.100) at 00:0c:29:ca:06:13 [ether] on eth1.2222
? (10.123.45.100) at 00:0c:29:ca:06:1d [ether] on eth1.2345
I have tried setting the ens224 in promiscuous mode, I have tried adding a routing entry to the gateway for that interface but that did not help.
I did manage to get it running by actually giving this ens224 interface an IP address but that is what I prefer not to have and which (if I understand the documentation correctly) is not needed.
Apparently I’m doing something wrong.
What is the correct way to configure this?
0 Answers