My challenge
We have Exchange servers at various sites, but also aboard ships. The ships are connected to our network through satellite links when at sea, but switch to WiFi bridges when in port.
Due the high latency (500+ ms) and not-uncommon drop-outs (e.g. when the ships are turning), attempting to send any e-mails above a few megabytes while at sea, is likely to fail and be retried until the limit has been reached. The result: The email doesn't get delivered and each try consumes valuable bandwidth on the sat link.
One "solution" is to limit the maximum e-mail size to say 5 MB, but that's hardly user friendly and an unnecessary restriction while in port.
Rough idea
What I'd rather do, is to queue all e-mails larger than a set limit for later delivery when at sea, while sending all small e-mails immediately. I was then thinking I'd ping the hub transport server in our datacenter regularly, when latency drops under ~400 ms, I'd start processing the large e-mails queue. When latency goes up over 400 ms, I'd plug the hole and let e-mails queue up again.
Now, I haven't gotten my hands really dirty with Exchange since version 2003. Back then, you could schedule large e-mails for later delivery, so my idea was do something similar in Exchange 2010, then script a way to switch the delivery schedule for large e-mails between 'always' and 'never'.
Obstacle
It shouldn't be too complicated to create a script like that, but then I read that the feature I'd rely on was removed with Exchange 2007:
This was a feature present in Exchange 2003 but has been removed for Exchange 2007. It was set on an SMTP Connector with the 'use different delivery times for oversize messages'.
TechCenter: Is it possible to schedule email delivery based on size in Exchange?
Questions
Is it true? - Is this feature no longer present in Exchange 2010, or has it merely transformed into something similar, I can use to accomplish my goal? If so, what?
Is there another way to defer delivery of large e-mails on certain Exchange servers? It could be based on a schedule or maybe even requiring specific action - I'm fairly certain there will be some way to trigger the delivery through script, I just need large e-mails in a separate queue on ships.
Your thoughts on this will be highly appreciated! :-)
Edit #1: Refined Rough Idea
I stumpled upon two PowerShell CmdLets I think can bring me pretty close to my goal:
I toyed around with Get-Message for a while, to see what kind of messages the commands above would deal with.
Most importantly, these commands accept a message size filter. This command will list queued messages,on the current server, larger than 5 MB (5,242,880 bytes):
get-message -Filter {Size -gt 5242880}
It seems Get-Message only returns messages from various remote delivery queues. But does messages flowing within the server, however briefly, show up in a queue that Get/Suspend/Resume-Message will mess with?
If not, the solution could be as simple as a scheduled script every few minutes, along the lines of (in pseudo code):
if ping_rtt > 400 Then
Suspend-Message -Filter {Size -gt 5242880}
Else
Resume-Message
EndIf
Concerns/follow-up questions:
Mostly irrelvant now - see edit #2.
Will Get-Message only return messages from remote delivery queues - never messages for intra-server delivery? If not, does the identity name of remote delivery queues follow a certain pattern, that I can use for filtering?
Could/should this be done via a custom Transport Agent (as suggested by @longneck) or an Event Sink (if this concept still exists in Exchange 2010)?
Say I run the script every 5 minutes, that still means large messages being sent, can potentially cause problems for up to 5 minutes, before getting suspended. We'd still be better off than we are now, but it's not optimal. I could increase the frequency to every minute, but it wouldn't be the most elegant solution.
Even if I only check round-trip time every 5 minutes (to save sat traffic), what Exchange mechanism would I need to setup, in order to check against the last recorded RTT, each time a message is submitted that goes to a remote delivery queue, and then take approriate action?
Edit #2: Proposed Solutions
Allow me to summarise the proposed solutions, and their pros and cons as I see it:
Custom Transport Agent
Concept
- Periodically monitor latency, classify as high or low (threshold: 400 ms?)
- Through a custom Transport Agent, suspend/resume all e-mails larger than a set threshold, when latency classification changes
- Through the custom TA, immediately put subsequently submitted large messages in "suspend" mode, if latency is high
Strengths
- Large e-mails are never attempted delivered when latency is high
Weaknesses
- No development skills to make this in-house (note to self: source code should belong to my company as part of the contract with the external developer)
- 3rd party software that ties into Exchange can cause problems when patching or updating
- Some sort of support agreement necessary, in case something goes wrong (see above)
Moderate Large Messages
Concept
- Periodically monitor latency, classify as high or low (threshold: 400 ms?)
- Based on latency classification, configure Exchange Transport Rules through scripting, to either let all messages flow or forward large messages to moderator
- Approve messages in moderator queue when ship's in port, possibly by a human
Strengths
- Large e-mails are never attempted delivered when latency is high
- Messages are suspended using native native Exchange Transport Rules
Weaknesses
- By the looks of it, messages can not be approved programmatically when latency is low, hence human intervention is required each time ship's in port
- Possibly privacy issues, if moderation is not handled programmatically
Questions
- Can messages be approved programmatically from moderator mailbox? How?
Scheduled PowerShell commands
Concept
- Periodically monitor latency, classify as high or low (threshold: 400 ms?)
- As long as latency is high, frequently (every minute?) suspend any large messages (
Suspend-Message -Filter {Size -gt 5242880}) - When latency drops to low, resume all messages (
Resume-Message)
Strengths
- Very simple to implement
Weaknesses
- Not the most elegant solution
- Delivery of each new large message can be attempted for as long as the interval between
Suspend-Messagecommands, possibly still wasting some bandwidth and create congestions (though very briefly compared to not doing anything)
Questions
- Any ideas on how to prevent attempts to deliver large messages, in-between
Suspend-Messagecommands? - Will
Get-Messageonly return messages from remote delivery queues - never messages for intra-server delivery? If not, does the identity name of remote delivery queues follow a certain pattern, that I can use for filtering?
Edit #3: The Way Forward
After bringing the proposed solutions up in my team (including the SMTP proxy, which I failed to include in edit #2), and based on my own gut feeling, we decided to go for a custom Exchange Transport Agent.
I'm in contact with a couple of consultancy companies, who will get back to me with how the will attack the problem and what it would cost.
If you have any experience with outsourcing programming tasks, feel free to leave feedback to my related question on Stack Overflow, because I don't.
Message Moderation
One probably non-ideal solution is to use a transport rule to forward messages over a certain size to either the sender's manager or a designated mailbox (on the ship's Exchange server) for moderation. This way if they are at sea, large messages can be essentially queued up in another mailbox. When the ship arrives at port, the manager or designated moderator can approve them for delivery.
The downsides are:
One upside is you would have a human making decisions on whether or not a message should even be sent such as if a user was trying to send a bunch of non work-related crap that would just bog down the connection for no reason. They could be corrected by administrative controls like pointing to a policy and slapping them on the wrist so they don't do it again.
Exchange 2010 Transport Rules
Custom Script
RE: A custom script to manage large e-mails. I could see something like the below that would enable/disable your Send Connector on the Exchange server. A downside is all mail is held in queue instead of just large messages, but some logic along these lines should work:
This logic isn't fully polished to make 100% sense, but you get the idea - queue all mail, check for latency, suspend large messages if it is high, un-queue mail, queue all mail, etc. Another downside of running a script like this is if it ever crashes or stops, how would you know to reinitialize it with no IT staff on board the ships? It could either potentially stop all outbound mail indefinitely (if it stops after disabling the Send Connector) or you could lose your large message control if it just leaves the Send Connector enabled and the script is no longer running.
SMTP Proxy
Even though you have indicated you are against any message handling outside of Exchange, After thinking about this some more, I've decided I'm with @longneck on using some type of SMTP proxy solution. Even IIS SMTP has a deferred delivery mechanism that it would seem Exchange 2010 does not. You could redirect large messages to the IIS SMTP server which could store them on disk, and, checking for latency first, have the IIS SMTP server send them via a script when latency is low. The worst-case if your scheduling mechanism got stuck or stopped would be the large messages get stuck on disk, but small messages continue to be sent. There are probably better solutions than IIS SMTP, and I have never used it myself, but it is just an example.
Configure SMTP E-Mail in IIS 7
Try to take a look to the new features of "Message Throttling" introduced with Exchange 2010 SP1. It could be very helpful for your use case.
http://technet.microsoft.com/en-us/library/bb232205(v=exchg.141).aspx
To solve the problem the way you have requested, you could write your own Transport Agent using the Microsoft Exchange Transport Agent SDK. Transport Agents are event based so Exchange will call a function in your library when a message is received. Your library can then do something like hold the message. If you don't have the skills in-house to do this, I'm sure you could hire a developer to write it for you.
But I don't think this is a great solution. As an alternative for you to investigate, you might want to look in to something like an SMTP proxy for low-quality links. As you have discovered, SMTP is a horrible protocol for low-quality links because an interrupted connection causes the message transmission to restart from the beginning instead of resuming from where it left off. If you're going to hire a developer for something, I would consider writing a server program that accepts incoming SMTP connections and sends the message on to a remote instance of this same program at the other end of the satellite connection in a resumable manner (possibly segregating the large messages from the small messages via TCP port to allow for QoS treatment by your WAN accelerator). The remote instance could then, upon receipt of the complete message, complete the delivery of the message via SMTP.