Home / server / Questions / 47079
In Process
Dan Carley
Asked: 2009-07-29 08:36:38 +0800 CST

Puppet Security and Network Topologies

  • 772

Background:

I am finally setting aside some time to join the 21st Century and look at Puppet.

As it stands today we version control all server configurations in a repository that is held internally at the office. When an update needs making, the changes are checked back into the repos and manually pushed out to the machine in question. This usually means SFTP'ing to the remote machine and then moving files into place, with the relevant permissions, from a shell.

So I am hopeful that Puppet is going to be an simple yet amazing extension to what we already have.

Now I consider the process that we currently have to be reasonably secure. On the assumption that our internal network will always be relatively more secure than the public networks in our datacentres.

  • The process is always one way. Changes traverse from a secure environment to insecure and never the other way round.

  • The master store is in the safest possible place. The risk of compromise, either by stealing configurations or sending out malicious modifications, is greatly reduced.

Question:

From what I understand of the Puppet server/client model is that the clients poll and pull updates down directly from the server. The traffic is SSL wrapped so cannot be intercepted or spoofed. But it differs from what we currently do because the Puppet server[s] would need to be hosted in a public location. Either centrally, or one for each datacentre site that we maintain.

So I am wondering:

  • Am I being unnecessarily paranoid about the change from push to pull?

  • Am I being unnecessarily paranoid about centrally storing all of that information on a public network?

  • How are others maintaining multiple networks - separate server for each site?

Update 30/07/09:

I guess that one of my other big concerns is placing so must trust in a single machine. The puppetmaster(s) would be firewalled, secured and such. But even so any public machine with listening services has an attack surface of a certain size.

Presumably if the master has permission to update any file on any one of the puppet clients, then it's compromise would ultimately result in the compromise of all it's clients. The "kings to the kingdom" so to speak.

  • Is that hypothesis correct?

  • Is there any way that it can be mitigated?

networking security puppet

7 Answers

  1. Because i sometimes store passwords in variables in my modules, to be able to deploy applications without having to finish configuration manually, it means that i can not decently put my puppet repo on a public server. Doing so would mean that attacking the puppetmaster would permit to gain some app or db passwords of all our different applications on all our servers.

    So my puppetmaster is in our office private network, and i do not run puppetd daemon on the servers. When i need to deploy, i use ssh from private net to servers, creating a tunnel and remotely calling puppetd.
    The trick is not to set the remote tunnel and puppet client to connect to the puppetmaster, but to a proxy that accept http connect and can reach the puppetmaster server on private network. Otherwise puppet will refuse to pull because of hostname conflict with certificates

    # From a machine inside privatenet.net :
ssh -R 3128:httpconnectproxy.privatenet.net:3128 \
    -t remoteclient.publicnetwork.net \
    sudo /usr/sbin/puppetd --server puppetmaster.privatenet.net \
    --http_proxy_host localhost --http_proxy_port 3128 \
    --waitforcert 60 --test –-verbose

    It works for me, hopes it helps you

    • 10

  2. We have two sites, our office and our colo. Each site has its own puppetmaster. We set up an svn repository with the following structure:

    root/office
root/office/manifests/site.pp
root/office/modules
root/colo
root/colo/manifests/site.pp
root/colo/modules
root/modules

    The modules directory under each site is an svn:externals directory back to the top level modules directory. This means that they share exactly the same modules directory. We then make sure that the vast majority of the classes we write are under the modules directory and used by both sites. This has the nice advantage of forcing us to think generically and not tie a class to a particular site.

    As for security, we host our puppetmaster (and the rest of our network) behind our firewall, so we're not that concerned about storing the config centrally. The puppetmaster will only send out config to hosts it trusts. Obviously you need to keep that server secure.

    • 4

  3. I can't make a judgment on how necessary your paranoia is, it highly depends on your environment. However, I can say with confidence that the two major points of your existing configuration can still apply. You can ensure your change traverse from a secure environment (the repository at your office) to the less secure environment, wherever your puppetmaster is located. You change the process from SFTP'ing to a bunch of servers and manually putting files in to place to SFTP'ing to your puppetmaster and letting Puppet distribute the files and put them in the correct place. Your master store is still the repository, and your risks are mitigated.

    I don't believe either push or pull are inherently safer than the other model. Puppet does a great job of securing the configurations in transit, as well as authenticating both client and server to ensure there is a two-way trust in place.

    As for the multiple networks - we handle it with a central "master" puppetmaster with sattelite puppetmasters at each location acting as clients to the central master.

    • 2

  4. One design approach is to have a puppetmaster local to each site of systems and use a deployment tool to push changes to the puppetmasters. (Using git with git hooks could work too).

    This would preserve your concern about listening services on a public network as the puppet network traffic would only be internal.

    It's also possible to push the manifests out to each server and have the puppet client parse the manifests and apply the relevant configs.

    • 1

  5. although you say "external", I really doubt arbitrary people need to connect to your puppetmaster. you can always throw a VPN into the mix. a friend of mine once asked me "do you need to worry about the security of the protocol if the connection is secure?" while I dont agree with that attitude, an extra layer never hurts and certainly works wonders on my personal paranoia. besides, its fun to tunnel tunnels.

    • 0

  6. Mark Burgess, the author of cfengine and a university professor (that puppet seems to owe its heritage to) has written alot about push and pull. He claims pull is inherently more secure. If you look at the cfengine website, they've only had like 1 network security incident in 17 years. Burgess claims that is because of the pull design. I think a single point of compromise is inevitable. I would be more concerned about the routes of attack to that point.

    • 0

  7. You can run puppet without a central master if you want. One method I've seen is using a git repository and having scripts that will only merge and deploy an update only if the tag is signed by one of a pre-set list of gpg keys. The people even worked out how to get stored configs (used for eg setting up nagios monitoring on a central server from a resource processed on another server).

    So if the central git server was compromised the other servers would not apply any more updates from it. The gpg keys would be on sys admin laptops or something, along with some way of revoking keys.

    Read more at http://current.workingdirectory.net/posts/2011/puppet-without-masters/

    • 0