Monitoring traffic via VXLAN for other systems

I am having difficulties getting VXLAN traffic to decapsulate correctly. Here is my setup: Ubuntu 18.04 systems with 2 NICs eth0 is connected to a live network and used for management eth1 is connected to an aggregated monitoring port that gets a copy of VXLAN traffic destined for multiple addresses, but is not joined to any real network

I can see the VXLAN traffic coming in. For example, here is a packet capture from eth1 on my system. You can see traffic for VNI 1 being sent to 10.196.17.65 and VIN 2 to 10.96.16.247:

IP 10.16.32.210.9092 > 172.24.45.166.25459: Flags [.], seq 117307907:117309237, ack 98470, win 115, length 1330
19:10:16.063986 IP 10.96.17.3.14698 > 10.96.17.65.4789: VXLAN, flags [I] (0x08), vni 1
IP 10.66.34.38.9997 > 172.20.154.49.53378: Flags [P.], seq 1:135, ack 475, win 38, options [nop,nop,TS val 935743084 ecr 2727174911], length 134
19:10:16.063988 IP 10.96.17.65.52704 > 10.96.16.247.4789: VXLAN, flags [I] (0x08), vni 2

My goal here is to get the decapsulated traffic for VNI1 and VNI2 onto the vxlan1 and vxlan2 interface.

The issue I am running into is that I don't get anything in vxlan1 or vxlan2 unless the networks for 10.196.17.65 10.96.16.247 are accessible to eth1, which they are not.

Here is an example for how I'm configuring vxlan1 and vxlan2:

ip link add vxlan1 type vxlan id 1 dev eth1 dstport 4789
ip link add vxlan2 type vxlan id 2 dev eth1 dstport 4789
ip link set vxlan1 up promisc on multicast on
ip link set vxlan2 up promisc on multicast on

Any thoughts here?