Fail2Ban on Ubuntu 10.04

Configuration files

/etc/fail2ban/jail.local

[DEFAULT]

ignoreip = 127.0.0.1
bantime  = 10 # made for test purposes
maxretry = 3

backend = polling

destemail = [email protected]


banaction = iptables-multiport

mta = sendmail

protocol = tcp

action = %(action_mw)s

[ssh]

enabled = true
port    = ssh
filter  = sshd
logpath  = /var/log/auth.log
maxretry = 3

[pam-generic]

enabled = true
filter  = pam-generic
port = all
banaction = iptables-allports
port     = anyport
logpath  = /var/log/auth.log
maxretry = 6

The rest of the fail2ban configs are just the default ones.

default /etc/pam.d/common-session-noninteractive

session [default=1]                     pam_permit.so
session requisite                       pam_deny.so
session required                        pam_permit.so
session required        pam_unix.so 
session optional                        pam_winbind.so 
session required        pam_loginuid.so 

changed /etc/pam.d/common-session-noninteractive

session [default=1]                     pam_permit.so
session requisite                       pam_deny.so
session required                        pam_permit.so
session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid
session required        pam_unix.so 
session optional                        pam_winbind.so 
session required        pam_loginuid.so 

Please note the only difference is adding session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid.

Logs

extract from /var/log/auth.log with default /etc/pam.d/common-session-noninteractive

May 22 15:30:01 node1 CRON[16029]: pam_unix(cron:session): session opened for user root by (uid=0)
May 22 15:30:01 node1 CRON[16029]: pam_unix(cron:session): session closed for user root
May 22 15:35:01 node1 CRON[16514]: pam_unix(cron:session): session opened for user root by (uid=0)
May 22 15:35:01 node1 CRON[16514]: pam_unix(cron:session): session closed for user root

Summary

1. If I execute fail2ban-client set ssh banip 1.2.3.4 in 15:26 the IP will get banned at 15:30. This is why I associate it with the above listed cron job.
2. If I modify /etc/pam.d/common-session-noninteractive and repeat the fail2ban-client command, got no entry in /var/log/auth.log and no ban.

More info:

1. default /etc/pam.d/common-session-noninteractive:

   fail2ban-client set ssh banip 1.2.3.4 -> the IP gets banned by an invisible cron job, which runs every 5 minutes. I checked every single file in /etc/cron* and /var/spool/cron/* and there was no such job present. Bottom line: the manual ban works with up to 5 minutes delay.

2. added session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid in /etc/pam.d/common-session-noninteractive as suggested here:

   fail2ban-client set ssh banip 1.2.3.4 -> the invisible cron job does not run and no ban happens.

My question:

how does the change in /etc/pam.d/common-session-noninteractive prevents the fail2ban-client to ban an IP? And why?

Edit

• Running in debug:

root@node1:~# fail2ban-client set loglevel 4
Current logging level is DEBUG
root@node1:~# fail2ban-client -vvv set ssh banip 1.2.3.4
DEBUG  Reading /etc/fail2ban/fail2ban
DEBUG  Reading files: ['/etc/fail2ban/fail2ban.conf', '/etc/fail2ban/fail2ban.local']
INFO   Using socket file /var/run/fail2ban/fail2ban.sock
DEBUG  OK : '1.2.3.4'
DEBUG  Beautify '1.2.3.4' with ['set', 'ssh', 'banip', '1.2.3.4']
1.2.3.4
root@zap:~# tail -f /var/log/fail2ban.log
2013-05-24 21:32:07,695 fail2ban.comm   : DEBUG  Command: ['set', 'ssh', 'banip', '1.2.3.4']
2013-05-24 21:32:07,696 fail2ban.filter : DEBUG  Currently have failures from 1 IPs: ['1.2.3.4']
2013-05-24 21:32:07,696 fail2ban.filter : DEBUG  Currently have failures from 1 IPs: ['1.2.3.4']
2013-05-24 21:32:07,696 fail2ban.filter : DEBUG  Currently have failures from 1 IPs: ['1.2.3.4']

Result: No Ban.

• Removing quiet from session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid in /etc/pam.d/common-session-noninteractive:

Result: Successful Ban.

/var/log/auth.log:

May 24 22:00:01 node1 CRON[22483]: pam_succeed_if(cron:session): requirement "service in cron" was met by user "root"
May 24 22:00:01 node1 CRON[22483]: pam_succeed_if(cron:session): requirement "service in cron" was met by user "root"

/var/log/fail2ban.log:

2013-05-24 21:56:07,955 fail2ban.comm   : DEBUG  Command: ['set', 'loglevel', '4']
2013-05-24 21:56:20,155 fail2ban.comm   : DEBUG  Command: ['set', 'ssh', 'banip', '1.2.3.4']
2013-05-24 21:56:20,156 fail2ban.filter : DEBUG  Currently have failures from 1 IPs: ['1.2.3.4']
2013-05-24 21:56:20,156 fail2ban.filter : DEBUG  Currently have failures from 1 IPs: ['1.2.3.4']
2013-05-24 21:56:20,156 fail2ban.filter : DEBUG  Currently have failures from 1 IPs: ['1.2.3.4']
2013-05-24 22:00:01,079 fail2ban.filter : DEBUG  /var/log/auth.log has been modified
2013-05-24 22:00:01,079 fail2ban.filter.datedetector: DEBUG  Sorting the template list
2013-05-24 22:00:01,853 fail2ban.filter : DEBUG  /var/log/auth.log has been modified
2013-05-24 22:00:01,853 fail2ban.filter.datedetector: DEBUG  Sorting the template list
2013-05-24 22:00:01,870 fail2ban.actions: WARNING [ssh] Ban 1.2.3.4
2013-05-24 22:00:01,870 fail2ban.actions.action: DEBUG  iptables -n -L INPUT | grep -q fail2ban-ssh
2013-05-24 22:00:01,876 fail2ban.actions.action: DEBUG  iptables -n -L INPUT | grep -q fail2ban-ssh returned successfully
2013-05-24 22:00:01,877 fail2ban.actions.action: DEBUG  iptables -I fail2ban-ssh 1 -s 1.2.3.4 -j DROP
2013-05-24 22:00:01,919 fail2ban.actions.action: DEBUG  iptables -I fail2ban-ssh 1 -s 1.2.3.4 -j DROP
2013-05-24 22:00:01,920 fail2ban.actions.action: DEBUG  
2013-05-24 22:00:01,923 fail2ban.actions.action: DEBUG   returned successfully
...

Fail2Ban version

fail2ban 0.8.7.1-2~ppa7~lucid from here. The stock one (version 0.8.4) kept failing with:

"global name 'time' is not defined"

which prompt me to look for newer version.

I have Watchguard XTM22-W as my firewall. On it there are few 1-to-1 NAT policies through which I access local services (mainly SSH on different machines). All these machines have Fail2Ban installed and send everything to a central log server. All machines run Ubuntu and use rsyslog.

As I collect all addresses to be banned on one place, I would like to have a script, running on this Linux box, where I will login to the Watchguard and execute couple commands there. This way I will stop the bad traffic at the entry point.

The issue I have with this is how to make passwordless admin login for Watchguard? It doesn't seem to provide functionality for that. Another option will be to tweak it auto-block feature, so it will ban IP addresses who abuse SSH. But then - how would it know how many attempts were made?

Thanks!

About a week ago I experienced one very interesting situation. I had a workstation - old desktop with Asus P5LD2 motherboard and 4 x 1 GB non-registered DDR2 Kingston memory. That same machine was a victim of a power stroke quite some time ago, IIRC 12-14 months ago. At the time of the power stroke the PSU fried and the HDD died. I replaced both, ran tests, including memtest and everything seemed fine. The user was working happily on it, until one day last week when he found some recent data "corruption" in some of his files. I investigated the issue and managed to narrow it down to motherboard fault. However, the "data corruption" was rather interesting and reproducible:

• copying text files from local directory to another local directory and running diff between both versions, there was only 1 bit changed somewhere random in the file;
• this bit was always the 6th out of 8, viewed in hex text editor, i.e. hex 19 becomes hex 39;
• the issue was reproducible while accessing NFS mounts and local mounts. Same exact tests repeated from other clients produced no differences;
• while copying from this machine over the network with rsync -av the command failed with Corrupted MAC on input. Disconnecting: Packet corrupt;
• tried same MB, but different memory set - again differences;
• old memory set on another Asus P5LD2 MB - no differences;
• memtest ran for more than 24 hours - no single error reported.

Conclusion from the tests - the bit flipping occurs only on this exact machine, regardless of the memory set used and the data location (local or NFS).

Based on all my tests, the only components left in the equation are the motherboard and the CPU.

My question(s) are:

1. what causes the bit flipping and how exactly it happens?;
2. is there a way to detect it?;
3. how to test/probe for it, when memtest fails?

I still have the troublesome machine in-house and am willing to run any tests to learn more about this.

The OS is Ubuntu Lucid 10.04, 64-bit.

Edit I forgot to mention that most (if not all) capacitors on the MB where bended on top, instead of flat.

389-ds on Ubuntu 12.04 server up and running. Enabled Fine-grained password policies and User must change password after reset for the whole tree. Created test user afterwards.

Login from CentOS client: user gets prompted to change its password: You are required to change your password immediately.

Login from Ubuntu client: user logs in, no prompt.

Copied CentOS client configuration files to Ubuntu client, precisely /etc/pam_ldap.conf (on Ubuntu this is /etc/ldap.conf), /etc/nslcd.conf, /etc/openldap/ldap.conf (on Ubuntu /etc/ldap/ldap.conf) - no dice.

Both clients authenticate successfully, both can change user passwords.

All logins are terminal logins, no GUI involved.

PAM on both clients:

1. Ubuntu:

   • /etc/pam.d/common-account

     account [success=2 new_authtok_reqd=done default=ignore]
     pam_unix.so account [success=1 default=ignore] pam_ldap.so account requisite pam_deny.so account required
     pam_permit.so

   • /etc/pam.d/common-auth

     auth [success=2 default=ignore] pam_unix.so nullok_secure auth [success=1 default=ignore] pam_ldap.so use_first_pass auth
     requisite pam_deny.so auth required
     pam_permit.so auth optional pam_cap.so

   • /etc/pam.d/common-password

     password [success=2 default=ignore] pam_unix.so obscure sha512 password [success=1 user_unknown=ignore default=die]
     pam_ldap.so try_first_pass password requisite
     pam_deny.so password required
     pam_permit.so password optional pam_gnome_keyring.so

2. CentOS

   • /etc/pam.d/system-auth-ac

     #%PAM-1.0 auth required pam_env.so auth sufficient pam_fprintd.so auth sufficient pam_unix.so nullok try_first_pass auth requisite pam_succeed_if.so uid >= 500 quiet auth sufficient pam_ldap.so use_first_pass auth
     required pam_deny.so

     account required pam_unix.so broken_shadow account
     sufficient pam_localuser.so account sufficient
     pam_succeed_if.so uid < 500 quiet account [default=bad success=ok user_unknown=ignore] pam_ldap.so account required
     pam_permit.so

     password requisite pam_cracklib.so try_first_pass retry=3 type= password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok password sufficient pam_ldap.so use_authtok password required pam_deny.so

     session optional pam_keyinit.so revoke session required
     pam_limits.so session optional pam_mkhomedir.so session
     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional
     pam_ldap.so

   • /etc/pam.d/passwd-auth-ac

     #%PAM-1.0 auth required pam_env.so auth sufficient pam_unix.so nullok try_first_pass auth requisite
     pam_succeed_if.so uid >= 500 quiet auth sufficient
     pam_ldap.so use_first_pass auth required pam_deny.so

     account required pam_unix.so broken_shadow account
     sufficient pam_localuser.so account sufficient
     pam_succeed_if.so uid < 500 quiet account [default=bad success=ok user_unknown=ignore] pam_ldap.so account required
     pam_permit.so

     password requisite pam_cracklib.so try_first_pass retry=3 type= password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok password sufficient pam_ldap.so use_authtok password required pam_deny.so

     session optional pam_keyinit.so revoke session required
     pam_limits.so session optional pam_mkhomedir.so session
     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional
     pam_ldap.so

One difference is that on Ubuntu I do not have cracklib installed. I plan to do so later, now I am just testing.

I wonder if Ubuntu LDAP client joins Windows AD, how does it receive notifications for password expiration from it. It should be something similar but I can't figure it out.

How to make the Ubuntu client to honour/obey the password policies? Why I don't see the You are required to change your password immediately. prompt when login, given that the same config works with CentOS?

Thank you!

Happy holidays!

I would like to customize my password expiration warnings. I figured PAM reads them from somewhere, but can't find from where. The question is relevant for both Ubuntu/Debian and Fedora/RHEL/CentOS.

Where does PAM read Warning: your password will expire in X days from?

Thanks!

Here is my config:

• Supermicro X7DBU;
• 3ware 9650 RAID Controller - driver installed, I use it as main datastore;
• managing the hypervisor with VMware vSphere Client 5.0.0 on Windows 2003 Server;

I have read that under the Configuration Tab on the client, Under Health Status, I should be able to see more hardware information than is displayed on the screenshot below.

What I want is very simple - a way to monitor the hardware status of the server as well as the RAID array state. Now, I have no way to know if a drive has failed, if the temperature is high, etc. I managed to install 3ware RAID CLI (tw_cli) and possibly this would help, but it solves the problem only partially.

The best solution should be a free one, as I have only this server and no money to spend on it.

How I can approach this?

I am evaluating VMware vSphere Hypervisor (formerly known as ESXi). The server version is 5.0.0. My steps:

1. Open VMware vCenter Converter Standalone (tried versions 4.3 and 5.0);
2. Attempt to convert a Linux box;
3. Successfully contacted the source machine;
4. Successfully contacted the server;
5. Successfully set up my task options;
6. Successfully started the task;

And at 1% the task failed.

The error: Unable to obtain the IP address of the helper virtual machine. I have DHCP on my network, but just in case I reran the task with a static IP setup for the helper. There it dies after 3 hours and still on 1%. The error message was similar. During the conversion process, I am unable to ping the Converter Helper Server for some unknown reason. It is on the same network, same VLAN as the source Linux box and the ESXi server.

My questions are:

• how to fix this error?;
• is there a better/easier/faster method to convert physical Linux box to VMware VM?

Thank you!

I am playing with LDAP lately and trying to authenticate against ApacheDS. I am unable to setup my Ubuntu client to print the LDAP users list (getent passwd for instance). I found few tutorials on the web of how to setup the server side. I do that via the Apache Directory Studio and have no issues at all - the server is up and running, the Studio sees the users and groups I got there. I am using the example apache_ds_tutorial.ldif from Apache DS website. I found some LDAP client setup on the Web, all made for OpenLDAP. I tested that one too and successfully authenticate against it. However, when I try to set it up in similar way for my ApacheDS it won't work.

Does anyone know how I could setup my LDAP client for ApacheDS? Thanks!

I have this sorts of entries in my /var/log/auth.log:

Apr  3 12:32:23 machine_name su[1521]: Successful su for user1 by root
Apr  3 12:32:23 machine_name su[1654]: Successful su for user2 by root
Apr  3 12:32:24 machine_name su[1772]: Successful su for user3 by root

Situation:

• All users are real accounts in /etc/passwd;
• None of the users has its own crontab;
• All of those users are logged in the machine some time ago via SSH or No Machine - time varies from few minutes to few hours;
• no cron jobs are scheduled to run at that time, anacron is removed;
• I can see similar entries for other days and other times. The common part is the users are logged in when it appears. It does not appear during login, but some time afterwards.

This machine has similar setup with few others but it is the only one where I see these entries.

What causes them? Thanks

Edit: I managed to narrow it down. I believe it is caused by cron @reboot. The funny part is - it runs "something" only for the users logged in right before the reboot. I checked /var/spool/cron, crontab -u <username> -l, grep -r @reboot /etc /var and can't see anything.

How I could run cron @reboot manually?

I have an NFSv3 server and around 15 clients. I am looking for pros and cons enabling async on the server side. I have read about it, but it is still a bit unclear to me. I know it can lead to data corruption, if the server crashes in the middle of write operation. However, I also read that the client stores a cache of that same operation and can recover it, if needed. My questions are:

• What exactly would happen if my server crashes (i.e. would it lose pending-to-be-written data, would it corrupt the underlying filesystem, etc)?;
• What would happen if both the server and the client crash at the same time (i.e. power failure/fault and UPS failure to handle it)?;
• What if the server crashes, but I have RAID BBU. Would the server recover safely?;
• Is there any way to detect such a corruption (something similar to fsck maybe)?;
• What if the server shutdown gracefully by UPS? Will I have chances of data corruption then?;
• What do you guys use - sync or async?

All machines are Ubuntu OS 10.04.

I was trying to find similar question here to no available. I have read the NFS Home Page and took a quick look at Managing NFS and NIS, 2nd Edition book.

I got Munin to notify me with email if certain values goes above my thresholds. This works very well and I am happy with it. However, there is a small glitch. If any of my values is in critical or warning state and Munin sends me notification about it, then on the next Munin update run I again receive the same notification, regardless that there is no change in the status. I found out that Munin run updates by default every 5 minutes.

Is there a way to make/force Munin to send single emails only on status change (i.e. first email for critical values, second for back to normal, not critical-critical-...-critical-normal)?

I got Intel X520 and connected it to PCI-E x16. The card is recognized as 10 Gbit by ethtool:

 # ethtool eth5
Settings for eth5:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full 
                                10000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
        Link detected: yes

The switch also recognizes the link as 10 Gbit. I ran iperf tests from 12 different machines (2 times x 6). Every single connection is 1 Gbit, except the server one. So, I expect to see around 6 Gbit total (6 machines x 1 Gbit each). I got only total of 1 Gbit on the server side. My questions are:

1. am I doing something wrong?
2. how to properly test the new card?

The OS is Ubuntu Karmic, the MB is Supermicro X8DAi. Thanks!

EDIT: The switch is single Netgear GSM7328S-200NAS. Here is a diagram:

I test by running:

Server: iperf -s
All clients, simultaneously: iperf -c server_IP

I see values around 150-160 Mbps/client, which match to 1 Gbps link to the server. The IP I assigned to the new card was never used, so I suppose no ARP issues. However, all traffic is on one subnet, as the same clients use the old 1 Gbps card and IP of the server.

No virtualization here, just plain Ubuntu install, serving NFS mostly.

Update: after further tests, I found that the clients' ARP tables contain wrong entry for server's 10 Gbps NIC. The HW address listed there is the same as the server's 1 Gbps NIC. I deleted it and manually inserted the correct entry: arp -d server_IP_10g, arp -s server_IP_10g HW:address. After that I was unable to ping server_IP_10g.

How can I fix it without disabling the 1 Gbps NIC in the server?

UPDATE2: sysctl -w net.ipv4.conf.all.arp_ignore=1 sysctl -w net.ipv4.conf.all.arp_announce=2 does not work.

I have 4-ports Intel 82571EB Gigabit adapter. It uses e1000e driver. The test machine is running Ubuntu 9.10, clients 9.10/10.04. I tried link aggregation (mode 4) and adaptive load balancing (mode 6). The setup is as follows:

server == 4-port NIC == 4 cat6 cables == Linksys SRW2024 switch == 4 or more clients.

I am testing with iperf and TCP. For mode 4 I setup a LAG in the switch. For mode 6 - no. My tests are:

4 or more clients, all of which have 1 Gbps connections are iperf clients. They run simultaneously. The server already listens for these connections (I had iperf -s running there). The results are the same as the server is running 1 Gbps only and not 4 Gbps combined. If I use one client I get 1 Gbps. If I have 4 or 8 clients the bandwidth per client goes down to 250 or 125 Mbps (these numbers are for illustration purposes only).

Why I can't get 4 Gbps to and out of the server? How I can fix it?

Few days ago I noticed something rather odd (at least for me). I ran rsync copying the same data and deleting it afterwards to NFS mount, called /nfs_mount/TEST. This /nfs_mount/TEST is hosted/exported from nfs_server-eth1. The MTU on both network interfaces is 9000, the switch in between supports jumbo frames as well. If I do rsync -av dir /nfs_mount/TEST/ I get network transfer speed X MBps. If I do rsync -av dir nfs_server-eth1:/nfs_mount/TEST/ I get network transfer speed at least 2X MBps. My NFS mount options are nfs rw,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountproto=tcp.

Bottom line: both transfers go over the same network subnet, same wires, same interfaces, read the same data, write to the same directory, etc. Only difference one is via NFSv3, the other one over rsync.

The client is Ubuntu 10.04, the server Ubuntu 9.10.

How come rsync is that much faster? How to make NFS match that speed?

Thanks

Edit: please note I use rsync to write to NFS share or to SSH into the NFS server and write locally there. Both times I do rsync -av, starting with clear destination directory. Tomorrow I will try with plain copy.

Edit2 (additional info): File size ranges from 1KB-15MB. The files are already compressed, I tried to compress them further with no success. I made tar.gz file from that dir. Here is the pattern:

• rsync -av dir /nfs_mount/TEST/ = slowest transfer;
• rsync -av dir nfs_server-eth1:/nfs_mount/TEST/ = fastest rsync with jumbo frames enabled; without jumbo frames is a bit slower, but still significantly faster than the one directly to NFS;
• rsync -av dir.tar.gz nfs_server-eth1:/nfs_mount/TEST/ = about the same as its non-tar.gz equivalent;

Tests with cp and scp:

• cp -r dir /nfs_mount/TEST/ = slightly faster than rsync -av dir /nfs_mount/TEST/ but still significantly slower than rsync -av dir nfs_server-eth1:/nfs_mount/TEST/.
• scp -r dir /nfs_mount/TEST/ = fastest overall, slightly overcomes rsync -av dir nfs_server-eth1:/nfs_mount/TEST/;
• scp -r dir.tar.gz /nfs_mount/TEST/ = about the same as its non-tar.gz equivalent;

Conclusion, based on this results: For this test there is not significant difference if using tar.gz large file or many small ones. Jumbo frames on or off also makes almost no difference. cp and scp are faster than their respective rsync -av equivalents. Writing directly to exported NFS share is significantly slower (at least 2 times) than writing to the same directory over SSH, regardless of the method used.

Differences between cp and rsync are not relevant in this case. I decided to try cp and scp just to see if they show the same pattern and they do - 2X difference.

As I use rsync or cp in both cases, I can't understand what prevents NFS to reach the transfer speed of the same commands over SSH.

How come writing to NFS share is 2X slower than writing to the same place over SSH?

Edit3 (NFS server /etc/exports options): rw,no_root_squash,no_subtree_check,sync. The client's /proc/mounts shows: nfs rw,nodev,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountproto=tcp.

Thank you all!

I did touch /forcefsck && reboot on remote machine, using Ubuntu 9.10.

How I can find out if there were any errors corrected? How I can find out what the result of the fsck was?

Thanks

AFAIK the reasons for file system corruption are as follows:

• improper shutdown (hard reset);
• hardware failures (bad block on disk, bad disk controller);
• improper startup (mounting damaged file system);
• kernel errors (really like to test this one).

Questions:

1. Is there some other reason for file system corruption, which I missed?
2. How to artificially cause file system corruption - I know about dd, but is there something more than this?

I am curious about Linux, but probably this will apply to Windows as well.

Is there a way to see what files an user deletes during his/her daily work. I know about bash_history, but I wonder if there is something more than this. The question is about plain Ubuntu (presumably any Linux) server installation.

If a user runs rm -fr dir1 in its home directory, would there be a log of the event? Do I have a way to easily enable such a feature?

Edit: Can I find out before installing anything? Both answers are excellent!

Thanks

I have a server, which exports home directories over NFS. They are on software RAID1 (/dev/sdb and /dev/sdc) and the OS is on /dev/sda. I noticed that my %iowait as reported by top and sar are relatively high (compare to the rest of the servers). The values range between 5-10%, as for the other servers (which are more loaded than this one) the same as 0-1%. The so-called user experience drops when the %iowait reaches values above 12%. Then we experience latency.

I don't have any drive errors in the logs. I would like to avoid playing with the drives using the trial-and-error method.

How I can find out which device (/dev/sda, /dev/sdb or /dev/sdc) is the bottleneck?

Thanks!

Edit: I use Ubuntu 9.10 and already have iostat installed. I am not interested of NFS related issues, but more of how to find which device slows down the system. The NFS is not loaded, I have 32 threads available, the result of

grep th /proc/net/rpc/nfsd
th 32 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Edit2: Here is part of iostat -x 1 output (I hope I'm not violating some rules here):

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          45.21    0.00    0.12    4.09    0.00   50.58

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   21.00    0.00   368.00     0.00    17.52     0.17    8.10   6.67  14.00
sdb               0.00     6.00    0.00    6.00     0.00    96.00    16.00     0.00    0.00   0.00   0.00
sdc               0.00     6.00    0.00    6.00     0.00    96.00    16.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00   21.00    0.00   368.00     0.00    17.52     0.17    8.10   6.67  14.00
dm-2              0.00     0.00    0.00   12.00     0.00    96.00     8.00     0.00    0.00   0.00   0.00
drbd2             0.00     0.00    0.00   12.00     0.00    96.00     8.00     5.23   99.17  65.83  79.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          45.53    0.00    0.24    6.56    0.00   47.68

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     1.00   23.00    2.00   424.00    24.00    17.92     0.23    9.20   8.80  22.00
sdb               0.00    32.00    0.00   10.00     0.00   336.00    33.60     0.01    1.00   1.00   1.00
sdc               0.00    32.00    0.00   10.00     0.00   336.00    33.60     0.01    1.00   1.00   1.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00   23.00    0.00   424.00     0.00    18.43     0.20    8.70   8.70  20.00
dm-2              0.00     0.00    0.00   44.00     0.00   352.00     8.00     0.30    6.82   0.45   2.00
drbd2             0.00     0.00    0.00   44.00     0.00   352.00     8.00    12.72   80.68  22.73 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          44.11    0.00    1.19   10.46    0.00   44.23

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   637.00   19.00   16.00   432.00  5208.00   161.14     0.34    9.71   6.29  22.00
sdb               0.00    31.00    0.00   13.00     0.00   352.00    27.08     0.00    0.00   0.00   0.00
sdc               0.00    31.00    0.00   13.00     0.00   352.00    27.08     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00   20.00  651.00   456.00  5208.00     8.44    13.14   19.58   0.33  22.00
dm-2              0.00     0.00    0.00   42.00     0.00   336.00     8.00     0.01    0.24   0.24   1.00
drbd2             0.00     0.00    0.00   42.00     0.00   336.00     8.00     4.73   73.57  18.57  78.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          46.80    0.00    0.12    1.81    0.00   51.27

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00   16.00    0.00   240.00     0.00    15.00     0.14    8.75   8.12  13.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

What are the most relevant columns to look into? What values are considered unhealthy? I suppose await and %util are the ones I am looking for. In my opinion dm-1 is the bottleneck (this is the DRBD resource metadata).

Double thanks!

Edit3: Here is what my setup is:

sda = OS, no RAID. Devices dm-0 and dm-1 are on it, as the latter is a metadata device for the DRBD resource (see below). Both dm-0 and dm-1 are LVM volumes; drbd2 = dm-2 = sdb + sdc -> this is the RAID1 device, which serves the user home directories over NFS. I don't think this one is the bottleneck. No LVM volume here.

Currently I am reading SSD reviews and I wonder how much exactly I will benefit if I move the 24 GB swap from 7200rpm HDD to SSD. Does anyone implemented swap space on SSD? Is this generally good idea?

On a side note: I read that ext4 has much better performance if the journal is on SSD. Anyone with such a setup?

Thanks!

Edit: Here I will answer the questions posted: Occasionally, relatively rare I am hitting the swap. I know what the swap is for and that is better to get more RAM. When the server begins to swap its performance degrades (not a surprise). The idea is if I have few memory hungry processes running, to improve the overall system performance at that time, using SSD for swap, instead of slower rotational media. At the end - I want to be able to login faster and check the server state during swapping, instead of waiting on the login prompt. And of what I see SSD is cheaper per GB than RAM.

Would I have better server performance during swapping (as rare it is) using SSD compared to HDD? Where 10k or 15k rpm HDDs would rate in this scenario?

Thank you all for your quick and prompt answers!

I have a NIS master-slave setup and I would like to improve the password rules/complexity for it. Seems like if I introduce new rules to the NIS Master they are applicable only from there. What I mean: I want minimal password length of 9 characters. On the NIS master if I run 'passwd' it obeys this requirement. If I run 'yppasswd' it just go to the default 6 characters. If I use 'chage -d 0 user1' to force a user to change a password, the user is prompted only when login on the NIS Master. The user's old password is still good to login at NIS clients. All machines are running Ubuntu 9.10 or 10.04.

How I can strengthen yppasswd rules and make it warn users to change their passwords? Thank you all!