Home / server / Questions / 1165819
Accepted
Taavi Ansper
Asked: 2024-09-27 18:07:14 +0800 CST

Linux VRF-s - no ping reply

  • 772

I am trying to get VRF-s working in linux.

Backstory: I have skimmed through a lot of documents for configuring VRF-lite in linux kernel and I have a simple request for the VRF - to return a ping to me :)

I have read through all of these docs:

  1. https://docs.kernel.org/networking/vrf.html
  2. https://alexkaouris.medium.com/vrfs-quick-intro-f0f7191779d2
  3. https://netplan.readthedocs.io/en/0.106.1/netplan-yaml/#properties-for-device-type-vrfs
  4. https://www.dasblinkenlichten.com/working-with-linux-vrfs/
  5. https://interpip.es/linux/creating-a-vrf-and-running-services-inside-it-on-linux/
  6. https://docs.frrouting.org/en/stable-9.1/zebra.html#virtual-routing-and-forwarding
  7. https://people.kernel.org/dsahern/management-vrf-and-dns
  8. https://jerryxiao.cc/archives/1004
  9. https://stbuehler.de/blog/article/2020/02/29/using_vrf__virtual_routing_and_forwarding__on_linux.html
  10. https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-59/Layer-3/VRFs/Virtual-Routing-and-Forwarding-VRF/#frrouting-in-a-vrf
  11. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/assembly_starting-a-service-within-an-isolated-vrf-network_configuring-and-managing-networking#proc_configuring-a-vrf-device_assembly_starting-a-service-within-an-isolated-vrf-network
  12. https://www.linkedin.com/pulse/running-vrf-linux-gerardo-marciales

But none of these talk of the simple scenario for pinging.

Configuration:

Ubuntu 24.04 VM

# netplan configuration
root@as-test-ubuntu:~# netplan get
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      match:
        macaddress: "fa:16:3e:d6:d5:0c"
      addresses:
      - "192.168.40.15/21"
      dhcp4: false
      set-name: "ens3"
    ens4:
      match:
        macaddress: "fa:16:3e:ef:59:e7"
      addresses:
      - "192.168.2.60/24"
      dhcp4: false
      set-name: "ens4"
  vrfs:
    vrf-mgmt:
      interfaces:
      - ens3
      routes:
      - to: "default"
        via: "192.168.40.1"
      table: 1

Logs from tcpdump on the VRF interface

root@as-test-ubuntu:~# tcpdump -npe -i vrf-mgmt
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vrf-mgmt, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:03:23.373513 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 560, length 64
13:03:23.373559 42:b6:48:39:49:0f > 42:b6:48:39:49:0f, ethertype IPv4 (0x0800), length 98: 192.168.40.15 > 10.192.10.82: ICMP echo reply, id 44994, seq 560, length 64
13:03:24.397483 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 561, length 64
13:03:24.397535 42:b6:48:39:49:0f > 42:b6:48:39:49:0f, ethertype IPv4 (0x0800), length 98: 192.168.40.15 > 10.192.10.82: ICMP echo reply, id 44994, seq 561, length 64
13:03:25.421560 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 562, length 64
13:03:25.421607 42:b6:48:39:49:0f > 42:b6:48:39:49:0f, ethertype IPv4 (0x0800), length 98: 192.168.40.15 > 10.192.10.82: ICMP echo reply, id 44994, seq 562, length 64
13:03:26.445783 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 563, length 64
13:03:26.445834 42:b6:48:39:49:0f > 42:b6:48:39:49:0f, ethertype IPv4 (0x0800), length 98: 192.168.40.15 > 10.192.10.82: ICMP echo reply, id 44994, seq 563, length 64

Logs on the tcpdump on the underlying ethernet interface

root@as-test-ubuntu:~# tcpdump -npe -i ens3
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:04:44.269975 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 639, length 64
13:04:45.293675 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 640, length 64
13:04:46.317790 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 641, length 64
13:04:47.341587 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 642, length 64
13:04:48.299941 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype ARP (0x0806), length 56: Request who-has 192.168.40.15 tell 192.168.40.1, length 42
13:04:48.299972 fa:16:3e:d6:d5:0c > 00:09:0f:09:01:1b, ethertype ARP (0x0806), length 42: Reply 192.168.40.15 is-at fa:16:3e:d6:d5:0c, length 28
13:04:48.365754 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 643, length 64
13:04:49.391177 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 644, length 64
13:04:50.414066 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 645, length 64
13:04:51.438125 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 646, length 64
13:04:52.463347 00:09:0f:09:01:1b > fa:16:3e:d6:d5:0c, ethertype IPv4 (0x0800), length 98: 10.192.10.82 > 192.168.40.15: ICMP echo request, id 44994, seq 647, length 64

As you can see the packets get stuck somewhere between the VRF and the underlying ethernet.

I have not found a solution for this.

linux

1 Answers

  1. Best Answer

    I was able to get some improvements by adding specific IIF and OOF rules via ip rule to point to the correct table, but in the end I opted out of using VRF-s and used normal routing tables.

    • 0