I ditched shorewall because it seemed overkill for the simple, unchanging ruleset this seems to call for.
I have 2 bastion firewalls in the rack between the host & each ISP.
The host contains an enclosure-bound VM which load-balances the 2 WAN connections and provides routing for multiple networks.
Since it also hosts the bridges, the VM host has an IP on each subnet that's connecting each bastion firewall to their own WAN port on the VM router.
The VM host doesn't need either of these connections other than to be able to provide the router VM access to the physical ports- the VM router provides the host's gateway & DNS.
The goal is to define rules which cause all inbound traffic on eth(n) and vmbr(n) use VM-WAN-IP-foo as a gateway, and all outbound traffic on eth(n) and vmbr(n) use bastion-fw-IP-bar as a gateway.
Also for eth(x) & vmbr(x).
Originally I thought I could do this with:
iptables -A INPUT -i eth(n) -s 0.0.0.0/0 -d 0.0.0.0/0 -j ROUTE --gw foo(n)
iptables -A OUTPUT -o eth(n) -s 0.0.0.0/0 -d 0.0.0.0/0 -j ROUTE --gw bar(n)
and
iptables -A INPUT -i eth(x) -s 0.0.0.0/0 -d 0.0.0.0/0 -j ROUTE --gw foo(x)
iptables -A OUTPUT -o eth(x) -s 0.0.0.0/0 -d 0.0.0.0/0 -j ROUTE --gw bar(x)
But that didn't take, so I must not be understanding correctly what I see in the manpage.
When I asked in the #netfilter IRC room someone mentioned that 'ROUTE' was depreciated, and to use -mangle instead. I would have never guessed that mangling would be needed for something like this.
Am I mistaken that this should be simple & straightforward?
I have a feeling there's an easier way to go about it that I'm just not aware of, maybe someone can enlighten me.
EDIT:
To put it another way, the VM Host is hosting these connection points, but isn't allowed to use them.
It and anything else that might find it's way onto those interfaces has to go around through the VM Router to reach the VM Host.
Here's a before & after drawing of what I mean:
|-------| |-------|
| ISP-A |<--| | ISP-B |<--|
|-------| | |-------| |
| |
V V
|-----------| |-----------|
| BASTION-A |<--| | BASTION-B |<--|
|-----------| | |-----------| |
| |
| |
| |
VM Host | |
| | |
| | |
\|/ | |
V | |
| |
||=================\|/=====================\|/==========||
|| |---V---| |---V---| ||
|| | |<----\ /---->| | ||
|| | | \ / | | ||
|| | eth(n)| V |eth(x) | ||
|| | | | | | ||
|| | |<--| | |-->| | ||
|| |-------| | | | |-------| ||
|| | | | ||
|| | | | ||
|| | | | ||
|| |-------| | | | |-------| ||
|| | |<--| | |-->| | ||
|| | | | | | ||
|| |vmbr(n)| | |vmbr(x)| ||
|| | | | | | ||
|| |-->| |<--| | |-->| |<--| ||
|| | |-------| | | | |-------| | ||
|| | | | | | ||
|| | | | | | ||
|| | | | | | ||
|| | | | | | ||
|| | | | | | ||
|| | | | | | ||
|| | / / \ \ | ||
|| | V V V V |---| ||
|| | |---------------| | ||
|| | | | | ||
|| | | | | ||
|| | |-->| VM Host | | ||
|| | | | | | ||
|| | | | | | ||
|| | | |---------------| | ||
|| | | | ||
|| | | | ||
|| V V V ||
|| |-------------------------------------------| ||
|| | WAN LAN WAN | ||
|| | | ||
|| | VM Router | ||
|| | | ||
|| | | ||
|| | | ||
|| | LAN LAN LAN LAN SAN | ||
|| |-------------------------------------------| ||
|| ^ ^ ^ ^ ^ ||
|| | | | | | ||
|| | | | | | ||
|| | | | | | ||
|| , , , , , ||
|| .*^*, .*^*, .*^*, .*^*, .*^*, ||
|| ( ) ( ) ( ) ( ) ( ) ||
|| (( ) (( ) (( ) (( ) (( ) ||
|| ( .) ( .) ( .) ( .) ( .) ||
|| `~ `~ `~ `~ `~ ||
|| ||
|| ||
||======================================================||
|-------| |-------|
| ISP-A |<--| | ISP-B |<--|
|-------| | |-------| |
| |
V V
|-----------| |-----------|
| BASTION-A |<--| | BASTION-B |<--|
|-----------| | |-----------| |
| |
| |
| |
VM Host | |
| | |
| | |
\|/ | |
V | |
| |
||=================\|/=====================\|/==========||
|| |---V---| |---V---| ||
|| | |<----\ /---->| | ||
|| | | \ / | | ||
|| | eth(n)| V |eth(x) | ||
|| | | | | | ||
|| | |<--| | |-->| | ||
|| |-------| | | | |-------| ||
|| | | | ||
|| | | | ||
|| | | | ||
|| |-------| | | | |-------| ||
|| | |<--| | |-->| | ||
|| | | | | | ||
|| |vmbr(n)| | |vmbr(x)| ||
|| | | | | | ||
|| |-->| |<--| | |-->| |<--| ||
|| | |-------| | | | |-------| | ||
|| | | | | | ||
|| | //============================// | ||
|| | // // | ||
|| | // IP Tables Says: 'Detour' // | ||
|| | // // | ||
|| | //============================// | ||
|| | / / \ \ | ||
|| | V V V V |---| ||
|| | |---------------| | ||
|| | | | | ||
|| | | | | ||
|| | |-->| VM Host | | ||
|| | | | | | ||
|| | | | | | ||
|| | | |---------------| | ||
|| | | | ||
|| | | | ||
|| V V V ||
|| |-------------------------------------------| ||
|| | WAN LAN WAN | ||
|| | | ||
|| | VM Router | ||
|| | | ||
|| | | ||
|| | | ||
|| | LAN LAN LAN LAN SAN | ||
|| |-------------------------------------------| ||
|| ^ ^ ^ ^ ^ ||
|| | | | | | ||
|| | | | | | ||
|| | | | | | ||
|| , , , , , ||
|| .*^*, .*^*, .*^*, .*^*, .*^*, ||
|| ( ) ( ) ( ) ( ) ( ) ||
|| (( ) (( ) (( ) (( ) (( ) ||
|| ( .) ( .) ( .) ( .) ( .) ||
|| `~ `~ `~ `~ `~ ||
|| ||
|| ||
||======================================================||
I might be wrong, but aren't you basically doing static NAT? Try the SNAT target (it's in the mangle table).
I am not sure if you can actually define gateway for inbound traffic.