Questions[pause](server)

I'm running two guest machines under KVM. The host is running Arch Linux (kernel version 2.6.38.2), the guests are both CentOS 5.5 (kernel version 2.6.18). I have the networking set up as a bridge, more-or-less as follows (I'm not doing it by hand, of course -- it's scattered through the system startup scripts)

# /usr/sbin/brctl addbr br0
# /usr/sbin/brctl addif br0 eth0
# /sbin/ifconfig br0 192.168.3.30 netmask 255.255.255.0 broadcast 192.168.3.255 up
# qemu-kvm -hda <disk-image1> -m 2048 -vga std -vnc :3 -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup
# qemu-kvm -hda <disk-image2> -m 2048 -vga std -vnc :4 -net nic,vlan=0 -net tap,vlan=0,ifname=tap1,script=/etc/qemu-ifup
# cat /etc/qemu-ifup
#!/bin/sh
sudo /sbin/ifconfig $1 0.0.0.0 promisc up
sudo /usr/sbin/brctl addif br0 $1

The configuration mostly works. What's odd is that I get frequent network "pauses". Most of my interactions with the virtual machines are through ssh or a web browser. Often, particularly after idle times, I'll go to type something in the session or access a web page, and nothing will happen. Eventually, after 10-15 seconds, normal activity will resume without error. ssh keystrokes from the dead time go through in a rush. Web requests seem to be lost, but a quick "reload page" gets the result without delay. The virtual machine as a whole continues to run the whole time, as I can see by watching the VNC session to the console. The two guests are independent in this way. Sometimes both will be frozen at the same time, but sometimes one will be working fine while the other is frozen. There are no errors in dmesg at the time of the freeze, on the host or guest.

Any ideas, for solution, cause, or further diagnosis?

(supplemental info, 7/27)

Here's the output from 'ifconfig -a' on the host:

hope 5$ sudo ifconfig -a
br0       Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet addr:192.168.3.30  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::7271:bcff:fea2:f4c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:140947463 errors:0 dropped:0 overruns:0 frame:0
          TX packets:66981660 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:192776331275 (183845.8 Mb)  TX bytes:9469675717 (9030.9 Mb)

br0:1     Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet addr:192.168.3.31  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

br0:2     Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet addr:192.168.3.32  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth0      Link encap:Ethernet  HWaddr 70:71:BC:A2:F4:C4  
          inet6 addr: fe80::7271:bcff:fea2:f4c4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:141848164 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68852413 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:195831275369 (186759.2 Mb)  TX bytes:9559920997 (9117.0 Mb)
          Interrupt:20 Memory:d0300000-d0320000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:27197534 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27197534 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:127242745992 (121348.1 Mb)  TX bytes:127242745992 (121348.1 Mb)

sit0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

tap0      Link encap:Ethernet  HWaddr 4A:F0:CA:32:F5:88  
          inet6 addr: fe80::48f0:caff:fe32:f588/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:397512 errors:0 dropped:0 overruns:0 frame:0
          TX packets:668318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:38045826 (36.2 Mb)  TX bytes:750358170 (715.5 Mb)

tap1      Link encap:Ethernet  HWaddr 56:92:5E:DE:93:67  
          inet6 addr: fe80::5492:5eff:fede:9367/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:25635 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88846 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:1999805 (1.9 Mb)  TX bytes:17071986 (16.2 Mb)

the output from 'brctl show' on the host:

hope 6$ sudo brctl show
bridge name bridge id               STP enabled     interfaces
br0         8000.4af0ca32f588       no              eth0
                                                    tap0
                                                    tap1

the output from 'ifconfig -a' on one guest:

guest1# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:192.168.3.35  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:323061 errors:0 dropped:0 overruns:0 frame:0
          TX packets:210150 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:298541685 (284.7 MiB)  TX bytes:25958962 (24.7 MiB)
          Interrupt:11 Base address:0x4000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2305 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2305 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:481780 (470.4 KiB)  TX bytes:481780 (470.4 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

and the same thing on the other guest:

guest2# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:192.168.3.36  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:62641 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25718 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10789140 (10.2 MiB)  TX bytes:2012261 (1.9 MiB)
          Interrupt:11 Base address:0x4000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:182 errors:0 dropped:0 overruns:0 frame:0
          TX packets:182 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:25798 (25.1 KiB)  TX bytes:25798 (25.1 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

I’m running a Debian Squeeze AMD64 server. Target runlevel after boot is runlevel 2, which includes rsyslogd, cron, sshd and some other stuff, but not dovecot, postfix, apache2, etc. The system fails to reach runlevel 2 with several symptoms:

• The system hangs at trying to start rsyslogd
• Booting into runlevel 1 works, then login from the console works
• Starting rsyslogd from runlevel 1 via /etc/init.d/rsyslog hangs
• Starting runlevel 2 with rsyslogd disabled works
• But then, logging in via console fails: I get the motd, and then nothing
• Starting sshd from runlevel 1 succeeds
• But then, I cannot login via ssh. Sometimes password ssh login gives me the motd and then nothing, sometimes not even this. Trying to offer a public key seems to annoy the sshd enough to not talk to me any further.
• When rebooting from runlevel 1, the server hangs at trying to stop apache2 (which is not running, so this really should be trivial). Trying to stop apache2 when logged in in runleve 1 does hang as well.

And that’s just the stuff which fails all the time. RAM has been tested, dmesg shows no problems. I have no clue.

Update: (shortened) output from rsyslogd -c4 -d called in runlevel 1

rsyslogd 4.6.4 startup, compatibility mode 4, module path '' caller requested object 'net', not found (iRet -3003) Requested to load module 'lmnet' loading module '/user/lib/rsyslog/lmnet.so' module of type 2 being loaded conf.c requested ref for 'lmnet', refcount 1 rsylog runtime initialized, version 4.6.4, current users 1 syslogd.c requested ref for 'lmnet', refcount now 2

I can kill rsyslogd with Strg+C, then. /var/log shows none of the configured log files, though.

Update2: Thanks to @DerfK I still have no clue, but at least I narrowed down the problem. I’m now testing with /etc/init.d/apache2 stop (without an apache2 running, of course) which hangs as well and looks like an even more obvious failure.

After some testing I found out that a file with one single line:

/usr/sbin/apache2ctl configtest > /dev/null 2>&1

hangs, while the same line executed in an interactive shell works. I was not able to further reduce this line while, i. e. every single part, the stream redirections and the commando itself is necessary to reproduce the hang. @DerfK also pointed me to strace which gave a shallow hint about what kind of hang we have here:

• wait4(-1for the init scripts
• futex(0xsomepointer, FUTEX_WAIT_PRIVATE, 2, NULL for rsyslogd / apache2 binaries called by the init scripts

The system was installed as a Debian Lenny by my hoster in autumn 2011, I upgraded it to Squeeze immediately and kept it up to date with Squeeze, which then used to be testing. There were no big changes, though. I guess I never tried to reboot the system before.

Update3: I found the problem. My /etc/nsswitch.conf specified ldap as hosts lookup backup, which is not available at that time of the boot. Relying on dns solely fixes my boot problems.

I have made a batch script for backup recently. Somewhere in the middle of script I'll have to wait for some time to be reached and then resume the next line of the script. I've scheduled the script at 4:00PM and after the wait command the next line should start at exactly 5:30PM. I thought of using SLEEP command but it's not sure that the commands used before the wait command will end up at certain time(due to inconsistent file sizes) but it's sure that they will be done by 5:00 or 5:10 and next it should execute wait command which waits for certain system clock. I'm checking if there is any command that waits or sleeps until the time specified reaches the system time and resumes there after. Anybody came across this situation and how was that resolved?

I have a RHEL5 workstation that has recently started to "hiccup". About every thirty seconds, it apparently completely stops execution for about 4 seconds. Seemingly nothing runs during that period. Long term processes seem to catch up to their input, but new processes simply don't get started.

Concrete examples:

• I have this loop running in a shell:

  while date; do
     sleep 0.2
  done
  

  Output merely skips over the missing seconds:

  Fri Aug 13 15:20:29 EDT 2010
  Fri Aug 13 15:20:29 EDT 2010
  Fri Aug 13 15:20:29 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:34 EDT 2010
  Fri Aug 13 15:20:34 EDT 2010
  Fri Aug 13 15:20:35 EDT 2010
  Fri Aug 13 15:20:35 EDT 2010
  Fri Aug 13 15:20:35 EDT 2010
  

• If typing in a terminal, either local console or remote via ssh or telnet, echoback pauses during the unresponsive time, but catches back up when it starts responding again, with apparently no loss of input, just lag.

• pings go unresponded-to during the unresponsive time, but are responded to when it comes back:

  64 bytes from xxx: icmp_seq=1911 ttl=64 time=0.203 ms  
  64 bytes from xxx: icmp_seq=1912 ttl=64 time=0.199 ms  
  64 bytes from xxx: icmp_seq=1913 ttl=64 time=3202 ms  
  64 bytes from xxx: icmp_seq=1914 ttl=64 time=2196 ms  
  64 bytes from xxx: icmp_seq=1915 ttl=64 time=1197 ms  
  64 bytes from xxx: icmp_seq=1916 ttl=64 time=195 ms  
  64 bytes from xxx: icmp_seq=1917 ttl=64 time=0.201 ms  
  64 bytes from xxx: icmp_seq=1918 ttl=64 time=0.206 ms
  

  This would seem to imply that it is actually receiving input during the unresponsive period, as those ICMP packets are not being retransmitted.

• vmstat 1 output also delays, but does not catch up. It's almost as if those few seconds didn't happen. It also shows an uptick in waiting processes, and a downtick in interrupts and context switches:

  procs -----------memory----------  ---swap-- -----io---- --system-- -----cpu------
   r  b   swpd   free   buff  cache    si   so    bi    bo    in   cs us sy  id wa st
   0  0    132 3111220 305540 588012    0    0     0     0  1035  151  1  1  99  0  0
   0  0    132 3111096 305540 588012    0    0     0     0  1019  125  0  0  99  0  0
   0  0    132 3111220 305540 588012    0    0     0    44  1034  154  0  1  99  0  0
   1  0    132 3111096 305540 588012    0    0     0     0  1016  131  0  0  99  0  0
   6  0    132 3111096 305540 588012    0    0     0     0   417   82  0  0 100  0  0
   0  0    132 3111220 305540 588012    0    0     0     0  1041  155  0  1  99  0  0
   0  0    132 3111096 305540 588012    0    0     0     0  1019  123  1  1  99  0  0
   0  0    132 3111220 305540 588012    0    0     0     0  1032  142  0  1  99  0  0
   0  0    132 3111096 305544 588008    0    0     0    44  1019  134  0  0  99  0  0
  

Rebooting makes the problem go away for a while. This most recent time it took six days to come back. I'm not sure if that's consistent or not.

I had initially suspected that the problem might be related to the nVidia video driver module, but I shut down X Windows and removed the module, without change in the symptoms.

There is nothing in dmesg or /var/log/messages that seems remotely relevant or in any way coincides with the hiccups. It does not appear to be an issue with a hard drive, as I would expect iowait to be prominent during the unresponsive period if that were the case, but it's not. It feels unlikely to be a hardware problem, as the hiccups are pretty regular. I've been unable to time them down to milliseconds, but it's a pretty consistent 30/4/30/4/30/4.

Any ideas?

I have a script which creates a virtual machine and gives me back an IP address. Then I would like to do something like this:

waitforssh 192.168.2.38 && ssh 192.168.2.38

And it will wait for the machine to be up and ssh to be responding, then ssh into it.

waitforssh is the command I need to find.

Would nmap, netcat, fping or ping do the job? I tried netcat but it gives up in just a couple of seconds if the host is unreachable.

It needs to handle the fact that the machine itself is booting and might take some time to respond to network packets.