Emmel's questions

I'd like to be able to create a notification that alerts based on the availability of a group of services, instead of just one threshold. For example, say I have 10 AWS servers that all do the same thing, and I expect some of them to be overloaded / fail sometimes without hurting the application: I want Check_MK to notify me if 3 out of 10 of the servers (or higher) fail in a given service. If one fails, don't notify me. Another example, perhaps simpler: say you have an NFS mount point on 20 servers, from the same NFS server. I don't want to get 20 warnings or criticals when I can just get one.

The above examples in my environment are already grouped in service groups.

I tried three different mechanisms in Check_MK 1.2.6p16:

1. Using Business Intelligence. The grouping and warning setting was fantastic, it did what I wanted! But the Notification Rules don't allow for anything related to the BI components of the product!

2. Cluster - I set up a cluster for the AWS servers, but since some of my checks (most notably my HTTP active check) requires a hostname, that's not going to help. I don't think Cluster is the right rabbithole to go down here, but correct me if I'm wrong. I abandoned looking at that.

3. Service Group Alert -- the purpose of this original question. There's nothing in the Notifications logic that lets me alert on service group availability.

Anyone accomplish this with Check_MK?

So our DNS provider, every so often, experiences DDOS attacks on their systems that causes our front-facing web sites to go down.

What are some options in terms of reducing dependency on a SINGLE external managed DNS provider? My first thought was using lower expire TTL and other SOA TTLs, but it feels like these affect secondary DNS server behavior more than anything else.

i.e. If you experience a DNS outage (due to DDOS, in this example) that lasts more than, say, 1 hour, delegate everything to a secondary provider.

What do people do out there when it comes to their external DNS and using another managed DNS provider as backup?

Note to our friendly moderators: this question is much more specific then the " "generic mitigate DDOS attack" questions out there.

EDIT: 2016-05-18 (A few days later): So, first off thank you AndrewB for your excellent answer. I have some more information to add here:

So we reached out to another DNS service provider and had a chat with them. After thinking and doing a bit more research, it's actually a LOT more complicated than I thought to go with two DNS providers. This is not a new answer, it's actually more meat/info to the question! Here is my understanding:

-- A lot of these DNS providers offer proprietary features like 'intelligent DNS', for example DNS load balancing with keepalives, logic chains to configure how responses are handed back (based on geo location, various weights to records, etc. etc.). So the first challenge is to keep the two managed providers in sync. And the two managed providers are going to have to be kept in sync by the customer who has to automate interacting with their APIs. Not rocket science, but an ongoing operational cost that can be painful (given changes on both sides in terms of features and APIs).

-- But here is an addition to my question. Let's say someone did do use two managed providers as per AndrewB's response. Am I correct in that there is no 'primary' and 'secondary' DNS here as per spec? I.e., you register your four DNS server IPs with your domain registrar, two of them are one of your DNS providers, two of them are DNS servers of the other. So you would essentially just be showing the world your four NS records, all of which are 'primary'. So, is the answer to my question, "No"?