Home / server / Questions / 293862
Accepted
Rick Koshi
Asked: 2011-07-26 06:42:02 +0800 CST

Random temporary network freeze on kvm guest machines

  • 772

I'm running two guest machines under KVM. The host is running Arch Linux (kernel version 2.6.38.2), the guests are both CentOS 5.5 (kernel version 2.6.18). I have the networking set up as a bridge, more-or-less as follows (I'm not doing it by hand, of course -- it's scattered through the system startup scripts)

# /usr/sbin/brctl addbr br0
# /usr/sbin/brctl addif br0 eth0
# /sbin/ifconfig br0 192.168.3.30 netmask 255.255.255.0 broadcast 192.168.3.255 up
# qemu-kvm -hda <disk-image1> -m 2048 -vga std -vnc :3 -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup
# qemu-kvm -hda <disk-image2> -m 2048 -vga std -vnc :4 -net nic,vlan=0 -net tap,vlan=0,ifname=tap1,script=/etc/qemu-ifup
# cat /etc/qemu-ifup
#!/bin/sh
sudo /sbin/ifconfig $1 0.0.0.0 promisc up
sudo /usr/sbin/brctl addif br0 $1

The configuration mostly works. What's odd is that I get frequent network "pauses". Most of my interactions with the virtual machines are through ssh or a web browser. Often, particularly after idle times, I'll go to type something in the session or access a web page, and nothing will happen. Eventually, after 10-15 seconds, normal activity will resume without error. ssh keystrokes from the dead time go through in a rush. Web requests seem to be lost, but a quick "reload page" gets the result without delay. The virtual machine as a whole continues to run the whole time, as I can see by watching the VNC session to the console. The two guests are independent in this way. Sometimes both will be frozen at the same time, but sometimes one will be working fine while the other is frozen. There are no errors in dmesg at the time of the freeze, on the host or guest.

Any ideas, for solution, cause, or further diagnosis?

(supplemental info, 7/27)

Here's the output from 'ifconfig -a' on the host:

hope 5$ sudo ifconfig -a
br0       Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet addr:192.168.3.30  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::7271:bcff:fea2:f4c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:140947463 errors:0 dropped:0 overruns:0 frame:0
          TX packets:66981660 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:192776331275 (183845.8 Mb)  TX bytes:9469675717 (9030.9 Mb)

br0:1     Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet addr:192.168.3.31  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

br0:2     Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet addr:192.168.3.32  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0      Link encap:Ethernet  HWaddr 70:71:BC:A2:F4:C4  
          inet6 addr: fe80::7271:bcff:fea2:f4c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:141848164 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68852413 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:195831275369 (186759.2 Mb)  TX bytes:9559920997 (9117.0 Mb)
          Interrupt:20 Memory:d0300000-d0320000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:27197534 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27197534 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:127242745992 (121348.1 Mb)  TX bytes:127242745992 (121348.1 Mb)

sit0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

tap0      Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet6 addr: fe80::48f0:caff:fe32:f588/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:397512 errors:0 dropped:0 overruns:0 frame:0
          TX packets:668318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:38045826 (36.2 Mb)  TX bytes:750358170 (715.5 Mb)

tap1      Link encap:Ethernet  HWaddr 56:92:5E:DE:93:67  
          inet6 addr: fe80::5492:5eff:fede:9367/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:25635 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88846 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:1999805 (1.9 Mb)  TX bytes:17071986 (16.2 Mb)

the output from 'brctl show' on the host:

hope 6$ sudo brctl show
bridge name bridge id               STP enabled     interfaces
br0         8000.4af0ca32f588       no              eth0
                                                    tap0
                                                    tap1

the output from 'ifconfig -a' on one guest:

guest1# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:192.168.3.35  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:323061 errors:0 dropped:0 overruns:0 frame:0
          TX packets:210150 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:298541685 (284.7 MiB)  TX bytes:25958962 (24.7 MiB)
          Interrupt:11 Base address:0x4000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2305 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2305 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:481780 (470.4 KiB)  TX bytes:481780 (470.4 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

and the same thing on the other guest: 

guest2# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:192.168.3.36  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:62641 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25718 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10789140 (10.2 MiB)  TX bytes:2012261 (1.9 MiB)
          Interrupt:11 Base address:0x4000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:182 errors:0 dropped:0 overruns:0 frame:0
          TX packets:182 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:25798 (25.1 KiB)  TX bytes:25798 (25.1 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
linux networking kvm-virtualization bridge pause

2 Answers

  1. Best Answer

    I finally figured out the right answer to this one.

    Short answer: MAC address conflict.

    Long answer: For each virtual machine, qemu creates a virtual network card. It assigns a MAC address to this virtual NIC. By default, it assigns a fixed address (52:54:00:12:34:56) to every NIC. If you start more than one guest on the same host, they all get this same fixed address. You can even see this in the ifconfig outputs in my original question. Naturally, this is a problem if you bridge the guests together, since it's the same as having two physical machines with the same MAC address on the same net. ARP does its best and flips back and forth between them, but with long timeouts, giving me the results I was seeing.

    So, the solution is to add "macaddr=..." flags to the -net parameter for each guest, such that each one has a unique MAC address. Example:

    # qemu-kvm -hda <disk-image1> -m 2048 -vga std -vnc :3 -net nic,vlan=0,macaddr=52:54:00:00:00:03 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup
# qemu-kvm -hda <disk-image2> -m 2048 -vga std -vnc :4 -net nic,vlan=0,macaddr=52:54:00:00:00:04 -net tap,vlan=0,ifname=tap1,script=/etc/qemu-ifup

    In my opinion, qemu's default behavior should be smarter about this, but I suppose it's fairly important that each guest get the same MAC address every time it's started, regardless of the order they're started in. Perhaps the authors couldn't think of a way to ensure this.

    Regardless, if you're running multiple guests on the same network (virtual or otherwise), make sure you set the MAC addresses explictly and uniquely. Otherwise, you'll run into the same problem.

    • 5

  2. there two thing may be cause this problem 1. ldap 2. NetworkManager

    so solve this, disable ldap and NetworkManager

    • -1