policy routing to guests on other server

After long searching I've found the answer to my challenge, but the last step fails.

I have two identical servers that both serve multiple guests. The servers have two interfaces: one for the external IP's I got from my ISP and one for an internal back-end network. All guests get an internal back-end network address and are connected to the physical back-end network via a bridge. Both servers can ping all guests on both servers. I expose the guests by D/SNAT'ing the guest IP to one of the available IP's on the host. This works perfectly for guests that live on the host.

For the D/SNAT to work to the other host, I learned that I needed to add a route for packets that arrived on the back-end interface, because the default route told the stack on the other server to go back on the wrong interface and wrong public IP. I have this scenario working in two local virtual test machines, even over a "dummy" bridge to the physical back-end interface.

This also works on the real servers for DNAT'ing to the back-end IP address that lives on the back-end bridge but NOT for the addresses assigned to the guests on the other server, which are also connected to the same back-end bridge.

Configuration, server A:

/etc/iproute2/rt_tables:
..
200 internal

iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o $WAN_IF -j SNAT --to $PUBLIC_IP
iptables -t nat -A PREROUTING -i $WAN_IF -p tcp -d $PUBLIC_IP --dport 53 -j DNAT --to $GUEST_IP
ip rule add from $LAN_IP table internal
ip route add default via $LAN_GW dev $LAN_IF table internal

Where
WAN_IF = WAN bridge interface (br1)
PUBLIC_IP = is public ip address on WAN bridge if
LAN_IP = private ip on LAN bridge (br0)
LAN_GW = private ip on LAN bridge on other server

brctl show
br0             8000.003018a96c83       no              eth0
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
                                                        vnet4
                                                        vnet5
br1             8000.003018a96c84       no              eth1

When $GUEST_IP is ip of guest on same host, everything works. When $GUEST_IP is LAN ip of other server also. But I can't reach guests on other server?

Update

Adjusting the default gateway of the guest works, but that isn't a symmetric solution as I wanted to create. But if that needs to be, it's a small change to make things work, as long as I keep track of the routes. It would however render routing different protocols via different public IP's impossible though unless I create some rules for that on the guest as well.

Update 2

It turns out I can solve the problem by adjusting the default gateway of the guest alone, but this is not as 'flexible' as I had hoped for.

Is there a more elegant/flexible solution?