Questions[ipvs](server)

I am trying to load balance ISAKMP using linux load balancing. Google searches indicate others have used LVS to load balance udp services successfully. I cant seem to get the load balancer to rewrite the source IP/port no matter what I do.

I have tried using ldirectord as well as direct ipvsadm commands. I am trying to get this to operate in masquerade mode using source IP hash as the scheduling algorithm. All I see the LB doing is rewrite destination (correctly) to that of a real server but leaving the source IP/port alone. Naturally this breaks the protocol. Has anyone else tried anything like this ? Here is my ldirector config file: (not a production quality config. I am just trying to get the rewrites to work first)

checktimeout=25

checkinterval=1

autoreload=yes

logfile="/var/log/ldirectord.log"

virtual=A.B.C.D:500

 protocol=udp

 real=E.F.G.H:500 masq

 scheduler=sh

 checktype=on

Following an upgrade to v1.19.7 with kubeadm, my pods are unable to request the kube-dns service via the service's ClusterIP. When using the kube-dns pod IP address instead, DNS resolution works.

kube-dns pods are up and running:

$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7674cdb774-2m58h   1/1     Running   0          33m
coredns-7674cdb774-x44b9   1/1     Running   0          33m

logs are clear:

$ kubectl logs coredns-7674cdb774-2m58h -n kube-system
.:53
[INFO] plugin/reload: Running configuration MD5 = 7442f38ca24670d4af368d447670ad91
CoreDNS-1.7.0
linux/amd64, go1.14.4, f59c03d
[INFO] 127.0.0.1:40705 - 31415 "HINFO IN 7224361654609676299.2243479664305694168. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.003954173s

kube-dns service is exposed:

$ kubectl get svc  -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   301d

endpoints are also configured:

$ kubectl describe endpoints kube-dns --namespace=kube-system
Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=KubeDNS
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2021-01-19T14:23:13Z
Subsets:
  Addresses:          10.44.0.1,10.47.0.2
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns-tcp  53    TCP
    dns      53    UDP
    metrics  9153  TCP

Events:  <none>

here is my coredns ConfigMap:

$ kubectl describe cm -n kube-system coredns
Name:         coredns
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
Corefile:
----
.:53 {
    log
    errors
    ready
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Events:  <none>

on workers, kube-proxy is running:

$ kubectl get pods -n kube-system -o wide --field-selector spec.nodeName=ccqserv202
NAME               READY   STATUS    RESTARTS   AGE    IP              NODE         NOMINATED NODE   READINESS GATES
kube-proxy-8r65s   1/1     Running   0          78m    10.158.37.202   ccqserv202   <none>           <none>
weave-net-kvnzg    2/2     Running   0          6h3m   10.158.37.202   ccqserv202   <none>           <none>

networking between pods is working, as I am able to communicate between pods running on separate nodes (here, dnsutils runs on node ccqserv202, while 10.44.0.1 is the pod IP address from coredns-7674cdb774-x44b9, running on node ccqserv223).

$ kubectl exec -i -t dnsutils -- ping 10.44.0.1
PING 10.44.0.1 (10.44.0.1): 56 data bytes
64 bytes from 10.44.0.1: seq=0 ttl=64 time=2.101 ms
64 bytes from 10.44.0.1: seq=1 ttl=64 time=1.184 ms
64 bytes from 10.44.0.1: seq=2 ttl=64 time=1.107 ms
^C
--- 10.44.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.107/1.464/2.101 ms

I am using "ipvs" as kube-proxy mode (although I can confirm the exact same behavior happens when using "iptables" or "userspace" modes).

Here is my ipvsadm -Ln on node ccqserv202:

$ ipvsadm -Ln 
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.46.128.0:30040 rr
TCP  10.96.0.1:443 rr
  -> 10.158.37.223:6443           Masq    1      0          0         
  -> 10.158.37.224:6443           Masq    1      0          0         
  -> 10.158.37.225:6443           Masq    1      1          0         
TCP  10.96.0.10:53 rr
TCP  10.96.0.10:9153 rr
TCP  10.97.147.126:2746 rr
TCP  10.100.162.140:9000 rr
TCP  10.101.126.110:5432 rr
TCP  10.109.184.125:4040 rr
TCP  10.110.163.112:9090 rr
TCP  10.110.215.252:8443 rr
TCP  10.158.37.202:30040 rr
TCP  127.0.0.1:30040 rr
TCP  134.158.237.2:30040 rr
UDP  10.96.0.10:53 rr

as you can see, then are no realservers configured under 10.96.0.10 virtual addresses, but there are under 10.96.0.1 (which corresponds to the kubernetes API service).

I am able to open a connection to 10.96.0.1 on port 443

$ kubectl exec -i -t dnsutils -- nc -vz 10.96.0.1 443
10.96.0.1 (10.96.0.1:443) open

I am able to open a connection to 10.44.0.1 on port 53

$ kubectl exec -i -t dnsutils -- nc -vz 10.44.0.1 53
10.44.0.1 (10.44.0.1:53) open

it evens resolves!

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default 10.44.0.1
Server:     10.44.0.1
Address:    10.44.0.1#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

but this does not work when I use kube-dns ClusterIP 10.96.0.10

$ kubectl exec -i -t dnsutils -- nc -vz 10.96.0.10 53
command terminated with exit code 1

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default 10.96.0.10
;; connection timed out; no servers could be reached

here is dnsutils resolv.conf file:

$ kubectl exec -i -t dnsutils -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local xxxxx.fr
nameserver 10.96.0.10
options ndots:5

finally, when I try manually adding realservers to ipvs on the node,

$ ipvsadm -a -u 10.96.0.10:53 -r 10.44.0.1:53 -m

kube-proxy detects it and immediately cleans it:

I0119 16:17:27.062890       1 proxier.go:2076] Using graceful delete to delete: 10.96.0.10:53/UDP/10.44.0.1:53
I0119 16:17:27.062906       1 graceful_termination.go:159] Trying to delete rs: 10.96.0.10:53/UDP/10.44.0.1:53
I0119 16:17:27.062974       1 graceful_termination.go:173] Deleting rs: 10.96.0.10:53/UDP/10.44.0.1:53

also, we can see with tcpdump that DNS requests from dnsutils to 10.96.0.10 are NOT rewritten to 10.44.0.1 or 10.47.0.2 as they should be with ipvs

    10.46.128.8.53140 > 10.96.0.10.domain: [bad udp cksum 0x94f7 -> 0x12c4!] 4628+ A? kubernetes.default.default.svc.cluster.local. (62)
16:27:56.950950 IP (tos 0x0, ttl 64, id 45349, offset 0, flags [none], proto UDP (17), length 90)
    10.46.128.8.53140 > 10.96.0.10.domain: [bad udp cksum 0x94f7 -> 0x12c4!] 4628+ A? kubernetes.default.default.svc.cluster.local. (62)
16:27:56.951321 IP (tos 0x0, ttl 64, id 59811, offset 0, flags [DF], proto UDP (17), length 70)

a tcpdump on kube-dns pods on the other side shows that these requests never arrive.

I've now spent a full day trying to understand what is happening and how to fix, and I am now running out of ideas. Any help would be very much welcome.

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ unfortunately did not help.

Thank you!

tl;dr: DNS resolution in the Kubernetes cluster does not work when using the kube-dns service ClusterIP, although I am able to resolve when using the kube-dns pods IP address. I would think something is wrong with my kube-proxy configuration, but I can't find what.

In a VRRP configuration I've been using /32 subnet mask but I've never know the reason behind.

Sometimes I've using the same subnet mask as the primary interface, like /24, and in some cases it works and int others, until I set up /32, it doesn't.

For example, on Mikrotik documentation points to that, but without explanation:

Note: address on VRRP interface must have /32 netmask if address configured on VRRP is from the same subnet as on router's any other interface.

It's really a requirement? On FreeBSD CARP I didn't find a similar requirement.

I want to replace iptables(8) with IPVS for a TCP reverse proxy which involved dual-way NAT.

My current setup using iptables is functionally equivalent to a userspace forwarder (like socat(1)). It has the following setup:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8000 -j DNAT --to-destination 172.31.1.1:80
# iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o eth1 -d 172.31.1.1 -p tcp --dport 80 -j MASQUERADE

The machine has 10.0.0.2 on eth0 and 172.31.0.2 on eth1. It also has the following routes:

default via 10.0.0.1 dev eth0
172.31.0.0/24 dev eth1
172.31.1.0/24 via 172.31.0.1 dev eth1

I say this is "functionally equivalent" to a userspace proxy because all traffic coming to TCP address 10.0.0.2:8000 is forwarded to the intranet host 172.31.1.1, while on the target host the traffic appears to come from 172.31.0.2 instead of the real host. Neither side sees the actual address of the opposite, only this reverse proxy server.

I'm trying to learn IPVS by replacing this iptables-based setup with Linux Virtual Server. I started with this:

ipvsadm -A -t 10.0.0.2:8000
ipvsadm -a -t 10.0.0.2:8000 -r 172.31.1.1:80 -m

The critical point is, it's frequent that neither the source host nor the target host is reachable on the same link as one of the interfaces. Traffic may need to be sent to a gateway on both sides.

I couldn't figure out why curl http://10.0.0.2:8000/ on this host gives the expected result, while the same command doesn't even establish a TCP connection when run on a host outside of the network (10.0.0.2 / eth0 is public-facing).

Using tcpdump, I can see a SYN packet to the backend, a SYNACK packet returned, but no ACK afterwards (the 3rd step of TCP 3-way handshake).

I know that a VPN or just tunneling can solve this problem, but I want to do this without either of them.

In case it's relevant, this reverse proxy is running CentOS 7.7 with both SELinux and firewalld disabled.

I'm running two "services" that are served in linux kernel-space:

• Linux Netfilter Firewall ("iptables")
• Linux Virtual Server Loadbalancer ("IPVS", "LVS")

Now I want to (performance-)monitor my "application" like any other software I am running. What would that basically be? CPU-Time and Memory.

How do I get these out of the kernel? Memory consumption of iptables might be estimated by the size of the connection tracking table, same for ipvs. It's okay for me, but any hints are welcome.

But - what about the CPU time spent for processing Routing, Firewalling and Loadbalancing?

Does "system" time include processing of packet-forwarding and -filtering? What about ipvs-handling?