We are running several virtual machines using KVM with libvirtd with a routed network (the host systems routes an extra subnet for the virtual machines). Host and Guest OS is Gentoo (host on amd64, guests on x86). The VMs are multi-homed with an internal and external network.
A few weeks ago i started to notice sprodic connection issues to our mailserver: The connection timed out. After a few minutes, the connection worked again. During the outage, the internal network is OK, but no traffic goes through the external network (neither in, nor out).
A tracepath shows as the last hop the host. After it, only stars until the maximum hops (again, in and out).
However, if i run a mtr or something similiar from an external host, the connection always succeeds (but only from this host). Because of this, a suspect an arp issues or something like this. I tried to change the ageing time on the bridge, but without success.
Another interesting thing: Pinging another host on the internet from the virtual machine sometimes shows an icmp redirect warning with the IP of the ISP's gateway.
Until now, i have seen this problem only on this single virtual machine, but the setup is not so different from the others. Any ideas?
Edit: Here is the network layout:
Ok, i solved the problem: the problems where triggered by the ICMP redirect packets sent by the host system. I suspect a different routing path for the packets sent by the VM and the remote server caused the problem.
After setting
sysctl -w net.ipv4.conf.all.send_redirects=0, the host stops sending the ICMP messages, and all connection succeed.