Avada Kedavra's questions

Background: I am forced to remotely upgrade a server from Ubuntu 8.04 LTS to 10.04 LTS due to an incompability issue with the raid controller.

The internet connection to the server is somewhat stable and seldom drops. Despite that I am concerned about losing the connection over SSH while doing the upgrade, leaving the server in an unreachable state. I am also worried about the server not being able to boot after the upgrade, in case I will be unable to know what is the problem.

Action plan: What I am looking for is advice to minimize the risk of losing the server, I am aware that what I am doing is very risky. This is my current action plan:

1) Backup everything that matters, locally and externally.

2) Temporarily disable boot-time disk checks with fsck. (I will have no clue what is going on if the disk check would take a long time to finish). This would be done through fstab by changing the very last paramter from 1 to 0:

UUID=5b1ff964-7608-44fd-a38d-7e43ad6b4c11 /               ext3    relatime,errors=remount-ro 0       0

3) Starting all upgrade processes with with screen so that they can be resumed if I lose the connection. Ie:

sudo screen apt-get upgrade

Questions:

• Does my proposed action plan seem reasonable?
• Is disabling the boot-time disk check a bad idea?
• What else could be done to decrease the risk of losing the server?

Update: Almost all answeres suggested me to setup DRAC/IPMI which I have now done. This feels like a really great acheivement that will for sure make the risk much much smaller as I can follow the entire power cycle over KVM/console redirection. For future references, this is what I have done:

1) Installed ipmitool to setup IP address, gateway etc for IPMI v2.0:

sudo ipmitool lan set 1 ipaddr 192.168.1.99 
sudo ipmitool lan set 1 defgw ipaddr 192.168.1.1

2) Installed free-ipmi to change the NIC selection mode to shared (I have only one network interface connected to the network):

sudo ipmi-oem dell set-nic-selection shared 

3) Used DRAC's https interface on https://192.168.1.99 to launch the console redirection viewer. This allows me to follow the entire boot sequence as well as configuring BIOS, raid controllers etc. Awesome.

Update 2. Done. All went with a charm, took less than 30 min to do the job. I ended up not turning off the disk check as the redirected console gave me the freedom to interrupt it whenever I wanted to, but I let it run to the end.

Thank you guys, your wisdom is invaluable!

I am facing an incompatibility issue between Ubuntu 8.04 LTS (Hardy) and the PERC 6/i RAID controller on our Dell PowerEdge R710 servers. Various forum posts (Ubuntu forum) confirm that other users also have had problems with the 2.6.24 64 bit kernel. One user reports that the problems disappeared after upgrading to kernel 2.6.32.7 (manually compiled). Ubuntu website also states that the R710 is certified for 9.04 and 10.04 LTS which are running on kernels 2.6.28 and 2.6.32.

My question: Can I upgrade the kernel only, without upgrading the entire Ubuntu release, and is this recommended? Or, is the alternative, to upgrade to Ubuntu 10.04 LTS a better solution?

Please be aware that the update must be applied to both test and production servers, where the downtime and risk of failure of the production servers must be kept to a minimum. Also, if possible I would like to avoid compiling my own kernels since this is not recommended by the Ubuntu team.

My current kernel is:

uname -a
Linux es 2.6.24-24-server #1 SMP Tue Jul 7 19:39:36 UTC 2009 x86_64 GNU/Linux

Edit: apt-get only shows 2.6.24 releases of the kernel, so I can't just use apt-get to update, can I?

I sometimes find the following message in auth.log:

sshd[8888]: error: no more sessions

In sshd_config I have set "MaxStartups 300", and I shouldn't have nearly as many clients connecting concurrently. At busy times I guess that I could have about 50-60 concurrent connections which only lasts for a couple of seconds while up/downloading a small file over sftp.

I suspect that mayby some of the clients isnt't closing the connection as they should, and as the connections happen repeadiately and quite frequenty (each client connecting a few times per minute), it could be adding up over time. To troubleshoot this I want to be able to count the SSH sessions, but I cant think of any better way than to count the connections using netstat:

netstat -an | grep 123.456.789.abc:22 | wc -l

As this would be counting connections instead of sessions and I am unsure if there is a case where the session would live, but the connection is closed?

Is there any better way of counting the current connectins, or even better print the current state of all sessions?

Thanks in advance!

I'm having a problem with a Linux system and I have found sysstat and sar to report huge peaks of disk I/O, average service time as well as average wait time.

How could I determine which process is causing these peaks the next time it happen?

Is it possible to do with sar? Can I find this info from the already recorded sar files?

Output of sar -d, system stall happened around 12.58-13.01pm.

12:40:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
12:40:01       dev8-0     11.57      0.11    710.08     61.36      0.01      0.97      0.37      0.43
12:45:01       dev8-0     13.36      0.00    972.93     72.82      0.01      1.00      0.32      0.43
12:50:01       dev8-0     13.55      0.03    616.56     45.49      0.01      0.70      0.35      0.47
12:55:01       dev8-0     13.99      0.08    917.00     65.55      0.01      0.86      0.37      0.52
13:01:02       dev8-0      6.28      0.00    400.53     63.81      0.89    141.87    141.12     88.59
13:05:01       dev8-0     22.75      0.03    932.13     40.97      0.01      0.65      0.27      0.62
13:10:01       dev8-0     13.11      0.00    634.55     48.42      0.01      0.71      0.38      0.50

I also have this follow-up question to another thread I started yesterday:

• Sudden peaks in load and disk block wait

Hello superior server gurus!

I'm running a Ubuntu server that hosts an apache tomcat service along with an MySQL database. The server load is always close to zero, even during the busiest hours of the week. Despite that I am experiencing random hangups 1-2 times per week, where the entire server stops responding.

An interesting effect of this lockdown is that all cronjobs seems to be executed later than scheduled, at least that is what the timestamps in various system logs indicate. Thus it appears to me that it is indeed the entire server that freezes, not only the custom software running as a part of the tomcat service. The hangup normally lasts for about 3-5 minutes, and afterwards everything jumps back to normal.

Hardware:
Model: Dell PowerEdge R720, 16 cores, 16 GB ram
HDD-configuration: Raid-1 (mirror)

Main services: 
apache tomcat, mysql, ssh/sftp

#uname -a
Linux es2 2.6.24-24-server #1 SMP Tue Jul 7 19:39:36 UTC 2009 x86_64 GNU/Linux

Running sysstat I can see huge peaks in both average load and disk block waits that corresponds in time exactly to when customers has reported problems with the backend system. Following is a plot of the disk usage from sar with a very obvious peak around 12.30pm.

My sincere apologies for putting this on a external server, but my rep is to low to include files here directly. Also had to put them together since i can only post one link :S

Sar plots: http://213.115.101.5/abba/tmpdata/sardata_es.jpg

Graph 1: Block wait, notice how the util% goes upp to 100% at approx 12.58

Graph 2: Block transfer, nothing unusual here.

Graph 3: Average load, peaks together with graph 1

Graph 4: CPU usage, still close to 0%.

Graph 5: Memory, nothing unusual here

Does anyone have any clue on what could cause this effect on a system? As I earlier explained the only software running on the server is a tomcat server with an SOAP interface, to allow users to connect to the database. Remote applications does also connect to the server via SSH to pull and upload files to it. At busy times im guessing that we have about 50 concurrent SSH/SFTP connections and not more than a 1-200 connections over http (soap/tomcat).

Googling around I found discussions about file handles and inode handles, but I think these are normal for 2.6.x kernals. Anyone that dissagrees?

cat /proc/sys/fs/file-nr
1152    0       1588671
cat /proc/sys/fs/inode-state
11392   236     0       0       0       0       0

At the same time "sar -v" shows these values for the time of the hangup above, but the inode-nr here is ALWAYS very high compared to above.

12:40:01    dentunusd   file-nr  inode-nr    pty-nr
12:40:01        40542      1024     15316         0
12:45:01        40568      1152     15349         0
12:50:01        40587       768     15365         0
12:55:01        40631      1024     15422         0
13:01:02        40648       896     15482         0
13:05:01        40595       768     15430         0
13:10:01        40637      1024     15465         0

I have seen this on two independent servers running the same setup of hardware, OS, software, raid-configuration etc. Thus I want to beleive that its more software/configuration dependent then hardware dependent.

Big thanks for your time
/Ebbe

this is my first post to this forum which I found through the superb podcast "It Conversations" from StackOverFlow. I am quite in my role as server administrator for an exhibition center in London. Basically we have a central file and sql server to which roughly 40 stations connects to to upload/download data used/captured by a set of applications.

Over the last weeks we have experienced a few random hangups to our applications, and as it always happen to multiple applications simultaneously I do not believe that the applications are the source of the problem. We also monitor the network using Dartware Intermapper which indicates that all switches and stations on the network has been reachable during the downtime. Thus, its all pointing to the server.

I have been looking through all log files I can think of and the only thing so far that I have found suspicious is the following lines in the syslog which are from the time of one of the hangups:

Feb  6 17:14:27 es named[5582]: client 127.0.0.1#33721: RFC 1918 response from Internet for 150.0.168.192.in-addr.arpa
Feb  6 17:14:40 es named[5582]: client 127.0.0.1#32899: RFC 1918 response from Internet for 152.0.168.192.in-addr.arpa
Feb  6 17:15:01 es /USR/SBIN/CRON[1956]: (es) CMD (/home/es/apps/es/bin/es_checksum.sh)
Feb  6 17:16:06 es /USR/SBIN/CRON[2031]: (es) CMD (/home/es/apps/es/bin/es_checksum.sh)
Feb  6 17:21:00 es named[5582]: *** POKED TIMER ***
Feb  6 17:21:00 es last message repeated 2 times
Feb  6 17:21:07 es named[5582]: client 127.0.0.1#44194: RFC 1918 response from Internet for 143.0.168.192.in-addr.arpa
Feb  6 17:21:12 es named[5582]: client 127.0.0.1#59004: RFC 1918 response from Internet for 164.0.168.192.in-addr.arpa

I find a few lines of interesting lines here:

1) "RFC 1918 response from Internet for 150.1.168.192.in-addr.arpa". I see this a lot in the syslog. And basically everytime I do a nslookup for any of the computers in the cluster I get a new similar line in the syslog. I understand from google that this has to do with reverse lookup problems. But I do not know how that could effect the systems. Lets say that one of these lines appear every time one of the userstations connects to the server, which may happen several times a second. Could this possible cause a hangup of the entire server?

2) POKED TIMER, I have googled this quite a lot, but not found an explaination that I can relate to. What does this mean?

3) The timestamps, it seems like the entire server has stopped responding for several minutes. Normally there are many printouts to the syslog per minute on this server. Furthermore the CRON job is set to run once every minute. Which according to the log, hasent happened here.

OS: Ubuntu 8.04 Kernel: Linux 2.6.24-24-server x86_64 GNU/Linux. Hardware: Dell R710, RAID1, CPU: 2x XEON E5530. 16GB Memory. Average load is very low, and memory should not be a problem.

Please let me know if you need any additional information.

Best wishes