I operate a small mail server for my private emails, some friends who have websites and two NGOs. In total my server sends between 60 and 400 messages a day. Now a lot of these emails are personal mails, between two or more people who know each other. Occasionally (usually once or twice a week) there will be a mailing that goes out to "members" of one NGO, informing them what's new etc.
Now I have already moved off the "mass mailings" (about 100 recipients, all personally known and manually subscribed through a paper form) to mailgun.org.
I still get (and increasingly so), rejected messages. Especially big email providers like Gmail, Yahoo or Microsoft (hotmail, live.com, ...) just decide to reject with a 550 or send personal messages to the Spam folder of the recipients. Sometimes this happens:
- gmail user sends email to user on my system
- user on my system replies
- the reply is being rejected or sent to spam
Things I have done:
- set up DKIM (per-domain signing of all outgoing email)
- set up SPF, domains usually have
~all, some-all - I have a correct PTR for my mail server IP
- obviously no open relay, users can only send from their own email address after authentication
- I have DMARC policies for most of the domains
- I rate limit outgoing messages, for some mail servers down to 1 per minute
- mail test services report "perfect" scores (all pass) for all of the above
- I regularily check my IP for blacklisting using http://www.dnsbl.info - it's always all green
Now the paradox comes here: for most of the big mail providers, there is a way to register to monitor rejection rates and IP reputation:
- https://postmaster.google.com
- https://postmaster.live.com/snds
- and I believe Yahoo has something similar
but I do not classify as bulk sender, because of the low volume. So I did register to monitor my reputation and rejection rates, but because I do not send bulk email, there are no reports.
Is there anything else I can do to improve mail delivery rates? Or should I give in and stop trying to operate my own mail server?
In case it is relevant: I use postfix and have very strict rules about incoming mail (i.e. no unknown domains/host names or invalid SPF records, I use spamassassin etc.)
Update
Here is an example, sent from me to my in-laws and it arrived in their SPAM folder: http://pastebin.com/BC6YgjpQ (I replaced the sending address domain with example.com and the receivers address with [email protected])
Since the question came up: Connections to Gmail are Untrusted TLS connection established to gmail-smtp-in.l.google.com[2a00:1450:400c:c0b::1b]:25: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits) encrypted.
There should be no issues becoming a small mail provider. You seem to be doing the right things. Many large providers don't get things right, and hopefully get most of their mail delivered.
If mail is being sent to the SPAM folder, it is likely you have missed something. There should be a record of why you have delivery issues:
A few things you did not specify although some should be caught by the validation report:
If you have DMARC you can configure delivery status reports and bounce reports. This will allow you to receive delivery reports. I receive reports from Google, Microsoft and Yahoo. Please note disposition "none" indicates the mail was delivered.
One thing missing in the above (excellent) replies is to set up outbound TLS. Gmail has started to punish senders not using TLS, and other providers aren't saying anything but I'm sure they will follow suit.
Nowadays, the spam activities are a real headache. The Big guys like Gmail, Microsoft, Yahoo etc. trying to secure their users from the spams. Hence, they must improve their techniques to filter the spams. And due to the security reason, they never disclose their Spam policies as well. Hence, we could not find a guideline to configure a mail server in such way that we can send mails to the big service providers.
There are no specific rules to be not listed in their bad book, but you should keep your server updated with the new guidelines. Here are some of them.
1) Check the root cause of the bounced back mail. Does it relate to the server IP reputation OR domain's incorrect DNS records.
2) Do not use an SPF record with a default value like ~all. Create a specific SPF record like a MX -all
3) Avoid mail forwarding from your server to Gmail/Yahoo/Microsoft/Comcast. If they detect any spam mail in your forwarded mails, they will not bother to check from where the mail is originated. They simply consider your mail server as a spam origin and you might be added to the blacklist.
4) Install an SSL on your server and use Outbound with TLS connection.
5) Keep Double Opt In list in all the newsletters. And many more...
I think you have a certificate chain error here. Make sure you are sending the intermediate certificate along with your certificate.
Answer is no.
I state this mostly because I'm aware that there are hard to imagine number of people with more experience in all aspects of being ESP admin than me and yet I run dozen of production email systems that qualifies as "small" without presented problems.
From what you have posted it looks like you have taught of everything and, presumed you have properly implemented what is listed, there is only one option left - that your IP address has done bad things in it's past life.
I mean (really) bad. Your IP address may be pulled from public blacklists at the ISP's initiative (which is vastly more efficient) but before that served virus and phishing traffic repeatedly for a prolonged period of time.
If that is the case, unfortunately there is little that can be done, to my best knowledge. One option is to remain in attempt to maintain legitimate email server for a long time, paying the price of many emails being rejected before sender reputation - completely different thing to being present or not on public blacklists - is slowly established.
Please keep us informed about your case.
@Stefan
You seem very knowledgeable which is awesome, I looked over your headers, without your domain name it makes it very hard to help troubleshoot. The one thing I did notice in your headers is that you're using "Simple/Simple" to sign your DKIM, you should really switch that over to "Relaxed/Relaxed". A lot of mail servers have trouble with simple.
You also left the mail server name in your pastebin, which does show up on a blacklist. I doubt this is causing your issue, but this tool does scan a lot more then the one you were using.
Send an email to
[email protected]to see how many criticals you have. You might get a few clues.To add to all the competent replies here already, our servers have recently had issues with emails not being delivered to destinations, and the issue was to do with the hosting service provider and completely outside of our control.
BACKGROUND
We rent servers from a national (UK) provider -- Specifically Fasthosts. They are in an ASN involving amongst others IONOS, Fasthosts, Arsys, 1&1 Mail and Media, Interdart and others.
Our servers are fully and correctly set up with PTR, DKIM, DMARC, SPF, etc. etc.
At semi-regular intervals our servers come up on the UCEPROTECT-Level3 spam listing, and for a couple of weeks every few months are marked incorrectly as spam.
WHY?
UCEPROTECT is one of the more prominent spam listing services and they are used by large, non-international mailing providers as one of many Blacklisting check services. They have levels 1- 2- and 3- listings. They do not provide a correction service for false listings, but they do allow server companies to pay them to be removed from their listings.
In discussion with our server hosts, Fasthosts, it came up that because neighbouring servers in our ASN blocks (123.456.789.XXX) are marked out on UCEPROTECT-3, then our servers are also marked out, simply by being in a nearby IP range in the same ASN . The offending servers may not even belong to the same Hosting Provider (probably belonging in this case to IONOS).
Fasthosts state they can't resolve the cause of the UCEPROTECT block because UCEPROTECT do not provide them enough information to do so. The most recent set of blacklistings come after a recent media advertising campaign by IONOS selling their small business server services.
CONCLUSION
It seems from a lot of digging, in the UK that many mid-level email providers are still using UCEPROTECT as a valid blacklisting service without realising that this misses a lot of genuine emails.
UCEPROTECT have a vested interest to keep blocking neighbouring IP blocks to encourage small providers to pay them to be whitelisted. It's a scam. But without a stepchange in the way medium-sized providers operate (ie they should ignore UCEPROTECT-3) there's nothing a small server host can do to escape the blacklist mark.