Læti's questions

Since a recent linux kernel upgrade (3.2.78 on Debian Wheezy) on both my NFS clients and NFS server, I'm seeing really high increase (10 to 20 folds) in the number of nfs_lookup requests from the clients to the server. This is making a huge performance degradation to the application running on my NFS clients (apache+php-fpm).

What can be the cause of such a high increase of nfs_lookup?

Output of mountstats --nfs on one of the clients after about 1h of uptime is as follows:

Stats for ruchbah.nfs://membres mounted on /srv/membres:
  NFS mount options: rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.74.13,minorversion=0,local_lock=none
  NFS server capabilities: caps=0x7fff,wtmult=512,dtsize=32768,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3
  NFS security flavor: 1  pseudoflavor: 0

Cache events:
  data cache invalidated 1231 times
  attribute cache invalidated 23693734 times
  inodes synced 1790 times

VFS calls:
  VFS requested 46296214 inode revalidations
  VFS requested 35900771 dentry revalidations

  VFS called nfs_readdir() 3678 times
  VFS called nfs_lookup() 35200320 times
  VFS called nfs_permission() 11201 times
  VFS called nfs_file_open() 22547 times
  VFS called nfs_file_flush() 1400 times
  VFS called nfs_lock() 7611 times
  VFS called nfs_fsync() 65321 times
  VFS called nfs_file_release() 3731 times

VM calls:
  VFS called nfs_readpage() 856 times
  VFS called nfs_readpages() 6945 times
  VFS called nfs_writepage() 10789 times
  VFS called nfs_writepages() 945872 times

Generic NFS counters:
  File size changing operations:
    truncating SETATTRs: 65272  extending WRITEs: 0
  291 silly renames
  short reads: 11201  short writes: 17
  NFSERR_DELAYs from server: 0

NFS byte counts:
  applications read 162084951 bytes via read(2)
  applications wrote 19636930 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 152501261 bytes via NFS READ
  client wrote 20783074 bytes via NFS WRITE

The number of VFS calls (nfs_lookup in particular) is very high. The cache attribute invalidation is also abnormally high. The clients mostly uses the NFS mount to read data.

We have an error in the %preun script of an RPM we distributed in the past. This has been corrected, but, now, upgrading to the newer version is failing because the previous version is not deinstalling cleanly:

Error in PREUN scriptlet in rpm package perl-perfSONAR-OPPD-MP-server
/var/tmp/rpm-tmp.veW9Jb: line 5: syntax error: unexpected end of file
error: %preun(perl-perfSONAR-OPPD-MP-server-3.4-1.pSPS.noarch) scriptlet
failed, exit status 2

This can be solved by uninstalling the outdated package manually without executing the %preun part and then reinstalling the new one:

rpm -e --nopreun perl-perfSONAR-OPPD-MP-server-3.4-1.pSPS.noarch
rpm -i --replacepkgs perl-perfSONAR-OPPD-MP-server-3.5-0.0.a1.pSPS.noarch.rpm

This is a manual procedure that requires intervention of the system administrator which we'd like to avoid.

Do someone know a way to do the same through yum so that the previous package is removed for the new one to get installed without errors, all that with a single yum command or through yum automatic upgrades?

We're using CentOS 6, if that matters.

Since the latest openssl upgrade on my Debian server, my mysql clients are unable to connect and give the following message

SSL connection error: error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small

I guess this is to prevent Logjam attacks.

Thanks to https://weakdh.org/sysadmin.html I know how I can generate stronger Diffie-Hellman parameters. However, I don't find any MySQL configuration option to actually use those.

Does anyone know?

I'm running a mail server for about 200 users with courier-imap on Debian 7. This server is running a linux kernel 2.6.38 inside a Xen guest having 2 cpu cores and 2 GB RAM. The server accept incoming mail through postfix, but resources intensive checks (amavis, spamassassin) are delegated to another server.

Lately I'm running into IMAP performance issues with client connections being dropped randomly. The load on the server is not low, but it is also not that high, even at times when those connection drops are happening. At peak times, I have about 120 simultaneous imap clients and 5 min load is < 1.

I've noticed the connection drops appear at the same time as the following log lines on my server:

imapd: FAMPending: timeout

I guess that when FAM is timing out, the imap server returns some kind of error and, depending on the client, the error is dealt with gracefully or not (some webmail clients do not like that).

The regular Debian courier-imap package requires a fam implementation, I have the choice of using famd or gamin. I've tried both and both are running into those timeouts. It seems to me gamin produces a lower load on the server, but still timeouts are occurring.

I'm wondering if there is anything I can tune to improve the fam/gamin performance and to avoid the timeout messages. There is not much to configure in the daemons. However, could it be that I'm running out of file descriptors or inotify resources? How could I check that?

Or do you think a better choice would be to upgrade my Xen guest or to move away from courier-imap?

On servers doing shared web hosting, or sharing the PHP environment between different PHP applications, I usually implement a security policy that uses PHP open_basedir to restrict each user to his/her own directory (along with other PHP directives like disable_functions for example). I also see quite a lot of posts here on SF discussing the benefits of using this feature.

However, now I stumble on this notice in the Debian package file README.Debian.security stating explicitly that they don't provide security support for (amongst others):

* Vulnerabilities involving any kind of open_basedir violation, as
  this feature is not considered a security model either by us or by
  PHP upstream.

So I wonder, does this statement is just there to waive any responsibility or does it have more fundamental reasons?

In particular, how would you go to secure a PHP server used by multiple different users without open_basedir and while trying not to raise the maintenance effort to high? Or would you just recommend to never do shared hosting as the PHP developers are stating in their security note?

One of my hosted server is a Xen guest running Debian and linux kernel 2.6.32 then 2.6.38. It has been running fine for quite some time (more than a year) but I recently noticed something weird. For a few months, the part of the memory allocated to buffers (disk I/O, network) always stays at zero.

The output of free is looking like that:

             total       used       free     shared    buffers     cached
Mem:       2071148    1775284     295864          0          0    1139388
-/+ buffers/cache:     635896    1435252
Swap:       524280      19508     504772

The buffers column always print 0, no matter the usage of the server (this is quite a busy mail server running postfix and courier).

Here is some more information:

• /proc/meminfo and /proc/slabinfo output: http://pastebin.com/raw.php?i=M8wSkj0S
• atsar -p (paging/swapping) output from a regular day: http://pastebin.com/VwLdf1L1 (it shows there is little swapping involved)

I also have the impression, but I cannot really measure it, that the system is slow at reading some files, files that are barely used or big files like logs. Hence my wondering if this buffers memory could be misconfigured in some way and that the disk throughput could be increased.

However, it seems there is no linux kernel parameter to govern that buffers memory, at least nothing like the parameters that exist for the cached memory. Or maybe I missed something?

How can this memory always be zero? Is this something I should worry about? How can I possibly change this behavior to put some RAM to better use if it is actually feasible?

Update: After a kernel upgrade (2.6.38 custom kernel from my provider), the behavior remains the same with buffers staying at 0. This is lasting for more than a year now, with the server being quite heavily used (mail server for about 2 hundred users). This server has been running Debian 6 and is now happily running Debian 7. Buffer memory still remaining at 0!

I have a Debian 6 Xen guest that seems to go to sleep from time to time. Randomly, it just stops to answer to any network requests (HTTP, ssh, ping) and only resumes activity when we log on the console. The server is clearly not crashed, however during this sleeping time no activity happens, even all the logs (syslogd and klogd) remain blank during this time.

Depending on when it happens and when we can actually log on the console, a few minutes but sometimes even an hour can be spent into this mode. This behavior happens irregularly, about once a month, randomly.

I don't have access to the console nor to the Xen host myself, but the support team from the hosting company is saying nothing suspicious is shown. They say it's the only guest on their infrastructure exhibiting this behavior.

The guest runs a linux 2.6.29.6 kernel compiled by the hosting company, has 2 cores, 4 GB of RAM and 2 GB of swap. 5-mins average load is not low (between 2 and 3, with peaks up to 5), but the swapping activity is low (swapin/swapout) and the swap space barely used. No kernel messages are spotted in the logs, nor in the dmesg output.

This server is running regular apache + mod_php and proftpd, really nothing fancy. AFAICT we've not tweaked any clock related parameters of the kernel (however I'm not sure how I can check the kernel setup if an energy saving mode or clock stepping is activated or not).

We're running out of clues at where the issue comes from.

Edit: I've run find /var -mmin -beforeevent -mmin +afterevent to try to find any file being modified during the last time the server hang and all that find reported was file being modified just before or just after the event, but nothing in between, even when it was a 1 hour long hang. This server only have a single partition, so it's not like only the disk containing /var was down.

I also have other hosts on the same subnet and all see this server as being offline: snmp polling fails and no requests are logged on the DB host from any PHP application running on the sleeping server.

We also tried to setup some cronjob to do continuous activity (like pinging some other host continuously), that didn't prevented this server to enter this sleep mode.