I have 2 machines, local1 and cloud1. local1 has 3 Nics: 2 connected directly to cable modems with public IPS (call them eth0 and eth1) and 1 connected to my LAN connection (eth2). cloud1 has a single nic (eth0) and is connected directly to the internet (1Gb/s link in a datacenter). eth0 and eth1 on local both have default gateways and send their traffic over either, depending on which IP is making the request. There are 2 OpenVPN P2P tunnels created, each using one of the public IPs from the modems, and connect to cloud1 over its public ip. This creates 2 tunnels, tun0 and tun1 on each box. they get ips 10.8.0.1/2 on tun0 (respectively) and 3/4 on tun1. 1 and 3 are on cloud, 2 and 4 are local.
the local box has its default route set to use both 1 and 3 (Debian 8.3 with whatever kernel is in the box) and that works "correctly" trace route shows me hitting both 1 and 3 at different times). but on cloud1, for it to get back to my local network (192.168.1.0/24) it has a route, but only through 10.8.0.2... nothing comes back over 10.8.0.4...
watching traffic monitors, i can see traffic going out over tun1 in house, but nothing (or very little) coming back in, while tun0 has lots of traffic coming both in and out...
I know how to set multiple default gateways using linux, but how do i set multiple non default gateways: for example:
ip route add 192.168.1.0/24 via 10.8.0.2 (works)
ip route add 192.168.1.0/24 via 10.8.0.4 (tells me it already exists)
Network forwarding is set on both boxes and i would like to do this without NAT or Masquerading... Also, 192.168.1.0/24 is a DMZ Network, so there are further upstream firewalls for other machines.
So, for anyone reading, i have managed to solve this. Zebra's Static Routes which allows me to have 2 routes for the one subnet. I had to remove the original route on the cloud box, and then zebra took over and all traffic is sent between those 2, equally... happy days!