Questions[sla](server)

I work for a small development house who are increasingly being asked to put together formal SLAs for our products based on particular configurations.

From a development side of things I'm comfortable with this, however there's no point in my saying that we'll meet particular targets from a software perspective if they're not realistic from a hardware/platform perspective - the clients only care about overall system availability.

What should I be looking at from a platform perspective? What sort of metrics and levels?

Also, what are the gotchas (for instance from a software perspective I'd never commit to a fix time - I have no idea whether I'm going to have to rewrite the whole product to correct something so saying that we can fix it in 5 days is potentially impossible - what should I avoid committing to from a hardware/OS/platform point of view)?

Greetings,

I'd like to ask the collectives opinion and view on distributed monitoring systems, what do you use and what are you aware of which might tick my boxes?

The requirements are quite complex;

• No single point of failure. Really. I'm dead serious! Needs to be able to tolerate single/multiple node failure, both 'master' and 'worker' and you may assume that no monitoring location ("site") has multiple nodes in it, or are on the same network. Therefore this probably rules out traditional HA techniques such as DRBD or Keepalive.

• Distributed logic, I would like to be deploying 5+ nodes across multiple networks, within multiple datacentres and on multiple continents. I want the "Birds Eye" view of my network and applications from the perspective of my customers, bonus points for the monitoring logic not getting bogged down when you have 50+ nodes, or even 500+ nodes.

• Needs to be able to handle a fairly reasonable number of host/service checks, a la Nagios, for ballpark figures assume 1500-2500 hosts and 30 services per host. It'd be really nice if adding more monitoring nodes allowed you to scale relatively linearly, perhaps in 5 years time I might be looking to monitor 5000 hosts and 40 services per host! Adding on from my note above about 'distributed logic' it'd be nice to say:

  • In normal circumstances, these checks must run on $n or n% of monitoring nodes.
  • If a failure is detected, run checks on another $n or n% of nodes, correlate the results and then use them to decide whether criteria has been met to issue an alert.

• Graphs and management friendly features. We need to track our SLAs and knowing whether our 'highly available' applications are up 24x7 is somewhat useful. Ideally your proposed solution should do reporting "out of the box" with minimal faff.

• Must have a solid API or plugin system for developing of bespoke checks.

• Needs to be sensible about alerts. I don't want to necessarily know (via SMS, at 3am!) that one monitoring node reckons my core router is down. I do want to know if a defined percentage of them agree that something funky is going on ;) Essentially what I'm talking about here is "quorum" logic, or the application of sanity to distributed madness!

I'm willing to consider both commercial and open source options, although I'd prefer to steer clear of software costing millions of pounds :-) I'm also willing to accept there may be nothing out there which ticks all those boxes, but wanted to ask the collective that.

When thinking about monitoring nodes and their placement, bear in mind most of these will be dedicated servers on random ISPs networks and thus largely out of my sphere of control. Solutions which rely on BGP feeds and other complex networking antics likely won't suit.

I should also point out that I've either evaluated, deployed or heavily used/customized most of the open source flavours in the past including Nagios, Zabbix and friends -- they're really not bad tools but they fall flat on the whole "distributed" aspect, particularly with regards to the logic discussed in my question and 'intelligent' alerts.

Happy to clarify any points required. Cheers guys and gals :-)

We had a rather serious outage this past week affecting several services which put us out of our SLA with customers. Now that everything has been resolved, I am conducting a post-mortem review.

From this review, I would like to come up with an internal document that describes the outage, its effects, our response and the resolution. I want to come up with a fairly standard form for future reuse. I have included my thoughts below, but what other items should be included? If this were a security-related incident, what would you add?

• Summary Executive level summary of event.
• Affected Services
• Impact What was the impact on our users and SLAs? Was there a cost in dollar terms, missed transactions, lost customers, etc?
• Outage Duration For each affected service if there were variances
• Cause Including primary and secondary causes
• Resolution
• Timeline of events Notifications, contact with external vendors, customer notifications, responses, etc.
• Problems with our response Did things not go as planned with our response to the outage? Correct people notified? Did vendors meet their contracted obligations?
• Preventative measures to take How do we prevent this outage from occurring again or reduce its impact?
• Detection Method How well did we detect this outage and how do we improve detection in the future?
• Changes to make in future outage responses

Try to keep posts down to one item and explanation, and this post can be updated with the top voted answers.

I would like to find a website that provides example service level agreements (SLA). Anyone know where I can find good examples?

• SLA = Service Level Agreement
• OLA = Operating/Operational Level Agreement

What's the difference between the two? Or is there really a difference?