Auto-Retry Config Push with Ansible or Saltstack?

I can only answer this for Ansible.

Ansible itself does not handle hosts which are not reachable. It will try to connect and once and if this is not possible the host is thrown out of the current play. But Ansible gives you some tools to deal with this yourelf.

First there is the wait_for module. With this you could wait with a very high timeout until the hosts are available.

- wait_for:
    port: 22
    delay: 10
    timeout: 3600
    host: "{{ inventory_hostname }}"
  delegate_to: localhost

This alone though would be a problem when you run the play, because Ansible by default would not process any further tasks until all hosts pass this task. Which is contra-productive in this case. According to your description the first hosts could be again unavailable when the last host finally was reachable.

For this to solve you need to use Ansible 2, which has a new feature called strategies. strategy: free allows you to run every task as fast as possible, which means it runs all tasks as soon as the host is available.

Still, a connection could go down and in this case there is no built-in way to automatically retry. If the ssh connection can not be established a fatal error will be thrown for this host and since Ansible ~1.9. there is no way to catch this kind of connection error. That does not affect other hosts though, they will all play fine.

You can retry though. Failed hosts will be stored in a file <playbook-name>.retry next to the playbook itself. To retry only failed hosts you then could run:

ansible-playbook ... --limit @<playbook-name>.retry

Salt runs in a pull model from the nodes to the master. You can issue global commands from the master like

salt 'api*.domain.com` state.highstate

That will run a highstate on all hosts that has a id(hostname) of api*.domain.com. A highstate is like a full chef run.

Usually by default people will either have the master schedule highstate runs on minions or they will run the schedule on the minions themselves to say run a highstate every 10 minutes.

So if a node is down and you run a command on the master to run a state then salt will report the node is down in its run output which can be formatted in many different ways for you to ingest. It can even be logged to mysql for example.

So for example if you ran the above command on the master server to run a highstate on all api*.domain.com nodes. If 2 of the 5000 were currently rebooting once salt-minion came back online they would get the even from the master via the message bus and run the highstate.

Salt also has a thing called proxy nodes to help the load of a master. You could have a single master somewhere and a proxy node in each datacenter and all the commands sent from the master go through the proxy nodes and the minions in those datacenters hit their proxy node and never the master

To extend Mike's answer, you can do push and pull simultaneously with Salt. Pushing is as easy as

salt 'api*.domain.com` state.highstate

At the same time, your minions can do scheduled pull every X minutes or hours via the built-in scheduler. My preferred method is to configure it via pillar, but adding it to the minion config works too. Something like:

schedule:
  highstate:
    function: state.highstate
    maxrunning: 1
    hours: 1
    splay: 600