Home / server / Questions / 558451
Accepted
John H
Asked: 2013-12-01 19:02:50 +0800 CST

Fixing intermittent file locking issue in Procmail

  • 772

Has anyone seen the following issue with procmail file lock timeouts when writing to a mbox file? This occurs every couple of weeks for a user with the following procmailrc:

:0 c:     #copy all mail to "bkp"
bkp

Most of the time, this works without issue. Every few weeks, the following message appears in the procmail log:

procmail: Forcing lock on "bkp"
procmail: Timeout, was waiting for "bkp"

Sometimes procmail is able to release the lock (or the lock disappears) before the postfix command timeout occurs (which is now set to an hour). Otherwise the mail delivery fails with:

relay=local, delay=2001, delays=0.78/0.05/0/2000, dsn=5.3.0,
status=bounced (Command time limit exceeded: "procmail -t -f-")

The bkp file is very large (over 10 gigabytes) but the issue is intermittent with several weeks in between instances of the problem and doesn't occur for other users who have the same procmailrc and gigabyte sized files (although none are as large as this).

The user would prefer not to use MailDir style folders and wants to keep this as a mbox file. Is there a way I can re-write the script to allow mail delivery to the user's mailbox while procmail waits on the lock for bkp? I've tried:

:0c      #copy all mail to "Saved"
{
 :0:
 bkp
}

EDIT: I've altered the above recipes from :0 w: to :0: since w waits on a program and none is being executed in this statement.

Which will allow two emails before it procmail stops accepting email again if I manually lock bkp during tests. If I release the lock before postfix times out, the mail will be delivered. I would also like to determine the root cause of the locking issue, but I haven't been able to trigger it yet except by locking the file myself. I have added LOCKTIMEOUT=10 to try and force the condition without success in normal mail delivery.

Here is the procmail version information:

procmail v3.22 2001/09/10 Copyright (c) 1990-2001, Stephen R. van den Berg Copyright (c) 1997-2001, Philip A. Guenther

Submit questions/answers to the procmail-related mailinglist by sending to:

And of course, subscription and information requests for this list to:

Locking strategies: dotlocking, fcntl() Default rcfile: $HOME/.procmailrc It may be writable by your primary group

timeout

2 Answers

  1. Have you considered using smaller daily/weekly backup mailboxes/files?

    You may use cron jobs to move them into main backup.

    man procmailex gives examples how to get destination names based on date output.
    man procmail gives example how to use formail to postproces mailbox file with locking.

    • 0
  2. Best Answer

    Here's what I did:

    • To test my issue, I built a vmware image of debian squeeze and limited the disk IOPS to 40 and memory to 256MB on the machine.

    • I used postfix, dovecot and procmail in the same way I had them set up on my production machine.

    • I created a user account and created large multi-gigabyte files for procmail to write to.
    • I installed nmon and sysstat to watch the disk usage I/O and tps values for the drive while Outlook and a local copy operation were running.

    What I found:

    • It was relatively easy to get Command Timeouts with postfix, but I still wasn't able to generate the locking error experienced by the user (except by using fcntl to lock the file).
    • I chose a 4 GB file to be my Save Archive, then ran "Repair Folder" on that IMAP folder in Outlook 2011 while copying a 1.3GB file on the server. This caused enough I/O on the server disk to generate timeouts in procmail.

    It isn't completely conclusive, but it looks like high disk load along with an outlook client accessing the mail file that procmail is attempting to lock can easily delay mail delivery long enough to hit the command timeout set in postfix.

    What I haven't tried:

    • I didn't re-compile the source and disable the locking as per FAQ 16 in the procmail source. In my case, the error is pretty rare and there is the work-around of rotating the Saved file regularly to keep it small.
    • 0