Home / server / Questions / 1003450
In Process
GAD
Asked: 2020-02-18 16:38:34 +0800 CST

Inter-Guest Network in KVM without a Bridge

  • 772

I would like to have a psuedo-wire between two guests in KVM. I cannot use a bridge of any type as the guest VMs are actually virtual switches themselves.

Conceptually, it seems that a veth-pair is the way to go, but after much reading I don't see a way to do this. I can put each guest into its own network namespace, and I can link two net-ns together using veth, but I can't get KVM to recognize the interfaces within the namespaces.

Any help is appreciated.

kvm-virtualization network-namespace

2 Answers

  1. A VM is usually handled with a tap interface, because its internal NIC is emulated by userspace (usually QEMU): on the other side of this interface presented by the kernel, there's a user space process handling traffic, while on the other side of a veth interface, there's yet again the kernel and its peer veth interface.

    Unless QEMU has a feature to connect directly those two interfaces (it's possible this feature does exist, but I didn't find it), like OpenVPN can connect two clients without involving the kernel network stack, then usually a bridge is expected to be used to connect them.

    If you can't use a bridge to move data from tap1 to tap2 and from tap2 to tap1 because it could interfere with bridge protocols, it's possible to use a lower level interface, working at the network interface level: tc and its tc ... mirred action which can move a packet from an interface to an other interface. Currently tc ... mirred can only have egress as direction for its action which is fine, that's the direction needed here to move ingress from one side to egress on the other side, as seen by the host. So Whatever VM1 outputs (to host's ingress), it's moved to VM2 (host's egress). Whatever VM2 outputs, it's moved to VM1.

              VM1                         HOST                        VM2

┌────────────────┐    ingress                     egress    ┌────────────────┐               
│                │      -----------redirect--------->       │                │
│ CPU == NIC == QEMU == tap1                     tap2  == QEMU == NIC == CPU │
│     or bridge  │      <----------redirect----------       │  or bridge     │
└────────────────┘    egress                     ingress    └────────────────┘

    If the tap interfaces are named tap1 and tap2, you can then "connect" them with this:

    tc qdisc add dev tap1 ingress
tc filter add dev tap1 ingress matchall action mirred egress redirect dev tap2

tc qdisc add dev tap2 ingress
tc filter add dev tap2 ingress matchall action mirred egress redirect dev tap1

    These commands create two half-duplex data transfer, for a full-duplex result. On older kernels matchall can be replaced with u32 match u32 0 0

    For real interfaces the interfaces should also be put in promiscuous mode to not filter by MAC, but I'm not sure that's really needed for virtual non hardware-accelerated interfaces. Else:

    ip link set tap1 promisc on
ip link set tap2 promisc on

    The host's network stack will not see redirected packets, as can be seen in this schematic: everything between ingress and egress is short-circuited. Tools like tcpdump will still capture those packets, since AF_PACKET is outside the short-circuit. But care should be taken to not have the host inject packets: interfaces should have IPv6 disabled (to avoid SLAAC, NDP, DAD etc.) and no IPv4 address assigned:

    sysctl -w net.ipv6.conf.tap1.disable_ipv6=1
sysctl -w net.ipv6.conf.tap2.disable_ipv6=1

    This will hardly interfere with any KVM usage (including libvirt), but it's related to interfaces. Those commands can be run only once the interfaces exist, and will have to be run again if they disappear and are recreated (VM stopped and restarted).

    You can get activity stats with:

    tc -stats filter show dev tap1 ingress
tc -stats filter show dev tap2 ingress

    If you have more than two VMs and must flood every flow to every other, it's possible, but filters and/or actions must be adapted to not be terminating and allow to first mirror (rather than redirect) as many times as needed and then do a final redirect action (or even no redirect if you want the host's network stack to see traffic).

    • 1

  2. While this is old, I still can suggest a solution.

    1. You can dedicate a physical NIC to the guest. For example, forward it as a PCI device.

    2. You can connect VMs with L2TPv3 or using various TCP and UDP raw tunneling ("socket") modes:

    -netdev socket,id=id[,fd=h][,listen=[host]:port][,connect=host:port]
-netdev socket,id=id[,fd=h][,mcast=maddr:port[,localaddr=addr]]
-netdev l2tpv3,id=id,src=srcaddr,dst=dstaddr[,srcport=srcport][,dstport=dstport],txsession=txsession[,rxsession=rxsession][,ipv6=on|off][,udp=on|off][,cookie64][,counter][,pincounter][,txcookie=txcookie][,rxcookie=rxcookie][,offset=offset]
    1. I can't find any info in recent manual, but I used socket in the following configuration in the past:
    qemu-system-x86_64  -name net-client-1 \
...
    -netdev socket,id=nioudp,udp=192.168.168.83:10004,localaddr=192.168.168.4:10083 \
    -device virtio-net-pci,netdev=nioudp,mac=08:00:27:62:B1:C1 \

    On the other side you have to reverse addresses and ports. It still works, however, it appears to be undocumented.

    For the details read the Networking section in the QEMU manual.

    • 1