Home / server / Questions / 220306
Accepted
grufftech
Asked: 2011-01-10 01:31:09 +0800 CST

Best Practice: Migrating Email Boxes (maildir format)

  • 772

So here's the situation.

I've got about 20,000 maildir email accounts chewing up a several hundred GB of space on our email server. Maildir by nature keeps thousands of tiny a****** little files, instead of one .mbox file or the like...

So i need to migrate all of these several millions of files from one server to the other, for both space and life-cycle reasons.

the conventional methods i would use all work just fine. rsync is the option that comes immediately to mind, however i wanted to see if there are any other "better" options out there.

Rsync not handling multi-threaded transfers in this situation sucks because it never actually gets up to speed and saturates my network connection, because of this the transfer from one server to another will take hours beyond hours, when it shouldn't really take more then one or two.

I know this is highly opinionated and subjective and will therefore be marked community wiki.

linux email maildir

2 Answers

  1. Best Answer

    consider a tar pipe like this one : 

    tar cf - . | ssh remote "cd /backup; tar xf -"

    I would also be looking at bbcp. I realise the page about bbcp is enormous and confusing, but take the time to read it because I feel its probably the best solution here.

    http://www.slac.stanford.edu/~abh/bbcp/

    I've done bbcp transfers on a variety of networks and found with the right thread settings you can normally saturate any network.

    Think carefully about the filesystem you are using as ext3 is likely to be very inefficient for lots of small files. I would be using XFS.

    • 3

  2. Depending on your service requirements, you might also want to consider using a combination of nginx's IMAP proxying and imapsync.

    Proxy the IMAP requests dynamically (using the authentication lookup to determine which backend server the user should connect to). Iterate over your user list with imapsync. Once the user has migrated to the new server, lock the account, run a final sync, and update a record somewhere so the auth lookup returns the new server.

    This way you generate no visible outage to the end-user.

    imapsync is also quite clever in how it does incremental syncs.

    I have done this to migrate users from Courier to Cyrus, and between mailstore backends.

    • 0