Getting NATed traffic returned back to the right interface

On a custom board (running a 4.x kernel), I have two physical ethernet interfaces and one radio modem that gives me a ppp-interface. iproute2 has been installed, and I am using nftables (not iptables).

My issue is that I want to forward specific UDP traffic (to ports 41000-41002) coming in on either ppp0 or eth0 to a specific host which sits behing eth1. The forwarding works as it should, but I can't get the return traffic to work correctly. Here's an example of how I have this set up:

eth0 has IP 192.168.10.10
eth1 has IP 192.168.100.10
ppp0 has IP 10.10.10.10

The external host which I want to send traffic to (and return traffic from), has IP 192.168.100.1, and it is physically connected to the same network as eth1.

Now, the nftables setup looks like this:

table ip firewall {
        chain incoming {
                type filter hook input priority 0; policy drop;
                ct state established,related accept
                iifname "lo" accept
                icmp type echo-request accept
                tcp dport { ssh} accept
        }

        chain prerouting {
                type nat hook prerouting priority 0; policy accept;
                iifname "ppp0" udp dport 41000-41002 dnat 192.168.100.1
                iifname "eth0" udp dport 41000-41002 dnat 192.168.100.1
        }

        chain postrouting {
                type nat hook postrouting priority 100; policy accept;
                oifname != "lo" masquerade
        }
}

My routing rules for iproute2 looks like this:

0:      from all lookup local
1000:   from 192.168.100.1 lookup ppproute
2000:   from 10.10.10.10 lookup ppproute
32766:  from all lookup main
32767:  from all lookup default

And the ppproute table only contains a default route over ppp0:

default dev ppp0  scope link

With this setup, everything works correctly if the inbound traffic comes in over ppp0. Packets are correctly rewritten by nftables/netfilter, and exits out eth1. The return traffic hits the 1000 rule, and exits back to the sender.

If the traffic comes in over eth0, it also hits the host at 192.168.100.1 as it should, but the return traffic is also routed back out over ppp0, which is my problem.

I have searched for a solution, and it has probably stared me in the face, but I haven't been able to solve this. I'm sure there is a simple way to fix this, so now I am hoping that someone would be so kind as to help me with this. I know I should have found a proper solution myself, but somehow I am missing it.