We have around 300 RHEL servers that are currently connecting to a Puppetmaster server. However, we have noticed some performance bottlenecks and it is the point of failure in our system. I am fairly new to puppet in general and I am considering creating a decentralized puppet architecture instead of having Puppet clients connect to the Puppetmaster server. Aside from what I would suspect to happen such as performance gain and lack of signing and exchanging SSL certs for new machines, what are other pros and cons to setting up a decentralized architecture?
Go decentralized.
Instead of signing certificates, make ssh keys. Don't give the keys to non-admins
You can use Git as your transport instead of subversion, and then you can branch for different machines/roles, and then version your changes, as well as allowing for... but you must know the DVCS spiel at this point.
It's faster, and less finicky to set up. Add some commit hooks for sanity checking.
Now, at this point, you've replaced the puppetmaster, with its client-server model, with ssh and git, both of which scale better than the puppetmaster.
Now, there might be a need in your organization for hierarchy. No problem, just store the git repo containing the definitive branch somewhere safe.
Bonus:
will allow you to see who made a change.
http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control
https://www.braintreepayments.com/braintrust/decentralize-your-devops-with-masterless-puppet-and-supply-drop?
Are you running puppet in passenger? stored configs? You really shouldn't have scalability problems at all with 300 nodes as long as you handle basic setup issues.
Decentralized is the best way to go cause each client compiles its own manifest from a local copy of the manifest source. This is updated each time you push updates from the say git server. Much more efficient use of bandwidth as clients don't ahve to contact the puppetmaster on every run. Also eliminates single points of failure as clients can be updated from anywhere.