I'm facing an issue with our server email. Yesterday, our email provider was down for over 20 hours, severely impacting our business operations.
We have approximately 5TB of archived emails, which users frequently search through for past contracts or other information. Despite the large volume, they insist on keeping all emails. The current setup is using postfix/dovecot, using mdbox format.
I've been tasked with improving the server's reliability and performance. My proposed solutions are:
Separate old and new emails: Move older emails to separate storage, reducing the sync load on the primary server.
Implement high availability: Add a new mail server in a different location. If one server fails, the other should take over and synchronize data upon the first server's recovery.
Introduce a mail proxy: Filter incoming emails and forward them to both servers for redundancy.
Leverage Amazon S3 for backup: Utilize the existing S3 bucket to store old emails, aligning with the date separation plan.
I'm considering using existing mdbox and rsync for these tasks, because we are currently using mdbox for mail storage, and rsync to backup all mail files and other files to the cloud storage. So, I am trying to use the existing methods. But I still have some questions:
Is simple rsync sufficient for syncing email files? Could it create duplicate entries if messages arrive at different times through the proxy?
How can I effectively separate email storage by year using mdbox?
Is there a more efficient tool for setting up Dovecot replication between two identical mail servers?
I have not been in touch with mail technology, and I would appreciate any recommendations or insights you can provide.
Thank you
It sounds like you’re dealing with a challenging situation. Improving server reliability and performance, especially with a large volume of archived emails, requires careful planning and execution. Here are some insights and recommendations for your proposed solutions:
You can separate emails by year using Dovecot's sieve filtering or scripts to move emails older than a specific date to a separate storage location. This can be accomplished by creating a script that periodically checks the email timestamps and moves older emails to a different directory or storage system. mdbox Management:
To manage mdbox more effectively, consider configuring Dovecot's mdbox_rotate_size and mdbox_rotate_interval settings to control when old emails are moved to secondary storage. 2. Implement High Availability Server Redundancy:
High availability can be achieved by setting up two servers with identical configurations and enabling Dovecot's built-in replication feature. This will ensure emails are synchronized between the two servers. Synchronization:
Utilize Dovecot's dsync tool for continuous replication. This is generally more efficient than using rsync for real-time syncing, as dsync is specifically designed for mail data synchronization. 3. Introduce a Mail Proxy Email Filtering and Distribution:
Consider setting up a mail proxy using software like HAProxy or Nginx. These tools can distribute incoming email traffic to both servers, ensuring redundancy. Potential for Duplication:
Be cautious with duplication when using rsync. It’s crucial to manage the mail queue properly to prevent duplicate entries. Setting up the mail proxy to handle queue management and deduplication logic can help mitigate this issue. 4. Leverage Amazon S3 for Backup Backup Strategy:
Using Amazon S3 for backup is a great idea, especially for storing older emails. You can integrate S3 with Dovecot using tools like s3ql or rclone to efficiently sync older email files to S3. Rsync Considerations:
While rsync can be used for regular backups, ensure it’s scheduled during low-traffic periods to minimize load on the servers. Consider using --delete and --archive flags for efficient backups without redundancy.