Home / server / Questions / 1097079
Accepted
E. Jaep
Asked: 2022-03-27 01:51:11 +0800 CST

Maximum throughput between Kubernetes pods with LACP bonding

  • 772

We currently have a kubernetes cluster (using calico) where all worker nodes are connected together with lacp bonds (2x10GB links with LACP Mode 4 - 802.3ad).

However, the maximum throughput we manage to get between 2 pods is 10 GB/s. According to some documentation, we should be able to achieve 20 GB/s. According to wikipedia, it seems to be a normal behavior:

This selects the same NIC slave for each destination MAC address, IP address, or IP address and port combination, respectively

This leads to understand that only one link is used for a given destination IP + Port combination.

My question is the following: Should we be able to reach a throughput of 20 GB/s between pods? Or is it normal that we are stuck at 10 GB/s?

kubernetes calico lacp

1 Answers

  1. Best Answer

    This is not specific to Kubernetes, this is normal behaviour of LACP. It does not provide true throughput increase, rather its action better be described as "deterministic distribution of connections" (not individual packets), and a fault tolerance.

    It extracts from the packets some header fields (determined by the mode) and hashes them. For instance, hash mode "layer3+4" takes OSI layer 3 and 4 information, e.g. IP and port. The hash directly determines which LACP leg to egress this packet. Whichever hashing mode you choose, all packets belonging to the same connection will be hashed to the same leg, so any single connection could not exceed a single leg throughput.

    When another connection appears, if you're lucky, it could utilize anoher LACP leg. In this case two connections will be distributed between legs and you will have twice total throughput between hosts. This is not guaranteed: it could happen they both be routed through the same leg. But, when you have many connections (as it usually is when we consider convergent clusters), on average both legs will be utilized.

    I can compare this to Kubernetes, if you wish. If you add nodes (and scale the deployment accordingly), you can increase the number of clients that could be served by the cluster. But you can't improve response latency (time to service) of a particular request by this scaling (if the cluster wasn't overloaded).

    • 3