We are trying to figure out if there is a way to create a stream/alert that will detect when an email address has been in a log more than X times. From what we can tell I can only count the total number of messages that match the stream.
For example, we would want to alert on a field called 'mailaddress' if the value has been the same more than 10 times in the past minute. Anyone have ideas on how to do this?
There are a number of tools available that are designed for scanning logs.
fail2banis one of the these. You would need to setup an expression to match and configure the appropriate action. This could include temporarily blacklisting the user on the firewall.You may catch poorly configured servers that have initial retry times configured in seconds rather than minute or hours. Spambots are likely to change their sending addresses frequently so you may miss them.
I've seen a number of bulk mailers retry at a fast rate using a different IP on each request. The corresponding domains, tend to be consistent on the first two or three levels.
I would fix duplicates being sent by fixing the mailing list data. You will have duplicates from temporary rejects that should be retried by your mail server. Use a reasonable initial retry like 1 hour and monitor your queue for entries that have been in the queue for a while. The domain part of an email address is always case insensitive, and the left site is almost always case insensitive.
Most mail servers will eliminate duplicate addresses on the same message. However, this doesn't help if messages are individualized.
My server will defer delivery for over an hour for every RFC violation I detect. This includes rDNS, ELHO name matching DNS, SPF and others. There are other reasons that a message will be delayed. The reason delivery acceptance was deferred should be logged.