Home / server / Questions / 1039310
In Process
Yarik Dot
Asked: 2020-10-20 08:12:26 +0800 CST

Mcollective/Choria sequential run

  • 772

Our setup: 1 application, tens of instances. Each instance is one of Baremetal/KVM or vmware VM/LXC container (depending on required size).

We update all instances via puppet (masterless), which is executed by mcollective. This works great.

The issue is high hard drive load, when puppet is executed on all instances at one time,

When we execute mco rpc puppetrun run, puppet is immediately executed on all machines in the cluster, which leads to hard drive overload due to unpacking deb packages.

We would like to run mcollective sequentially on limited number of servers at the same time.

We have updated the mcollective execution to to: mco rpc puppetrun run --batch XXX --batch-sleep YYY which helps, but it is not ideal as sometimes we deploy small changes, which are processed within a second and sometimes large changes with database migrations, which takes tens of seconds to perform.

We will be happy for a hint of how to approach this issue.

And also we would like to skip all solutions requiring external inventory as number of instances changes dynamically. Mcollective is great in this way.

mcollective

1 Answers

  1. One of the solutions (which I don't personally like) is currently considered:

    for i in $(mco ping); do
  mco rpc puppetrun run --with-identity $i
done

    This would run mcollective sequentially one by one.

    But I am looking for a better solution.

    • 0