We have around 300 RHEL servers that are currently connecting to a Puppetmaster server. However, we have noticed some performance bottlenecks and it is the point of failure in our system. I am fairly new to puppet in general and I am considering creating a decentralized puppet architecture instead of having Puppet clients connect to the Puppetmaster server. Aside from what I would suspect to happen such as performance gain and lack of signing and exchanging SSL certs for new machines, what are other pros and cons to setting up a decentralized architecture?
I am running a default configuration of Puppet on Debian Squeeze 6.0.4.
The server's FQDN is master.example.com. The client's FQDN is client.example.com.
I am able to contact the puppet master and send a CSR. I sign it using puppetca -sa but the client will still not connect.
The two machines have accurate date and time.
This is what appears in /var/log/syslog:
Apr 3 17:03:52 localhost puppet-agent[18653]: Reopening log files
Apr 3 17:03:52 localhost puppet-agent[18653]: Starting Puppet client version 2.6.2
Apr 3 17:03:53 localhost puppet-agent[18653]: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Apr 3 17:03:53 localhost puppet-agent[18653]: Using cached catalog
Apr 3 17:03:53 localhost puppet-agent[18653]: Could not retrieve catalog; skipping run
Here is some interesting output:
OpenSSL client test:
client:~# openssl s_client -host master.example.com -port 8140 -cert /var/lib/puppet/ssl/certs/client.example.com.pem -key /var/lib/puppet/ssl/private_keys/client.example.com.pem -CAfile /var/lib/puppet/ssl/certs/ca.pem
CONNECTED(00000003)
depth=1 /CN=Puppet CA: master.example.com
verify return:1
depth=0 /CN=master.example.com
verify error:num=7:certificate signature failure
verify return:1
depth=0 /CN=master.example.com
verify return:1
18509:error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error:s3_pkt.c:1102:SSL alert number 51
18509:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:
client:~#
master's certificate:
root@master:/etc/puppet# openssl x509 -text -noout -in /etc/puppet/ssl/certs/master.example.com.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 2 (0x2)
Signature Algorithm: sha1WithRSAEncryption
Issuer: CN=Puppet CA: master.example.com
Validity
Not Before: Apr 2 20:01:28 2012 GMT
Not After : Apr 2 20:01:28 2017 GMT
Subject: CN=master.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:a9:c1:f9:4c:cd:0f:68:84:7b:f4:93:16:20:44:
7a:2b:05:8e:57:31:05:8e:9c:c8:08:68:73:71:39:
c1:86:6a:59:93:6e:53:aa:43:11:83:5b:2d:8c:7d:
54:05:65:c1:e1:0e:94:4a:f0:86:58:c3:3d:4f:f3:
7d:bd:8e:29:58:a6:36:f4:3e:b2:61:ec:53:b5:38:
8e:84:ac:5f:a3:e3:8c:39:bd:cf:4f:3c:ff:a9:65:
09:66:3c:ba:10:14:69:d5:07:57:06:28:02:37:be:
03:82:fb:90:8b:7d:b3:a5:33:7b:9b:3a:42:51:12:
b3:ac:dd:d5:58:69:a9:8a:ed
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:FALSE
Netscape Comment:
Puppet Ruby/OpenSSL Internal Certificate
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Subject Key Identifier:
8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
X509v3 Extended Key Usage: critical
TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha1WithRSAEncryption
7b:2c:4f:c2:76:38:ab:03:7f:c6:54:d9:78:1d:ab:6c:45:ab:
47:02:c7:fd:45:4e:ab:b5:b6:d9:a7:df:44:72:55:0c:a5:d0:
86:58:14:ae:5f:6f:ea:87:4d:78:e4:39:4d:20:7e:3d:6d:e9:
e2:5e:d7:c9:3c:27:43:a4:29:44:85:a1:63:df:2f:55:a9:6a:
72:46:d8:fb:c7:cc:ca:43:e7:e1:2c:fe:55:2a:0d:17:76:d4:
e5:49:8b:85:9f:fa:0e:f6:cc:e8:28:3e:8b:47:b0:e1:02:f0:
3d:73:3e:99:65:3b:91:32:c5:ce:e4:86:21:b2:e0:b4:15:b5:
22:63
root@master:/etc/puppet#
CA's certificate:
root@master:/etc/puppet# openssl x509 -text -noout -in /etc/puppet/ssl/certs/ca.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha1WithRSAEncryption
Issuer: CN=Puppet CA: master.example.com
Validity
Not Before: Apr 2 20:01:05 2012 GMT
Not After : Apr 2 20:01:05 2017 GMT
Subject: CN=Puppet CA: master.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:b5:2c:3e:26:a3:ae:43:b8:ed:1e:ef:4d:a1:1e:
82:77:78:c2:98:3f:e2:e0:05:57:f0:8d:80:09:36:
62:be:6c:1a:21:43:59:1d:e9:b9:4d:e0:9c:fa:09:
aa:12:a1:82:58:fc:47:31:ed:ad:ad:73:01:26:97:
ef:d2:d6:41:6b:85:3b:af:70:00:b9:63:e9:1b:c3:
ce:57:6d:95:0e:a6:d2:64:bd:1f:2c:1f:5c:26:8e:
02:fd:d3:28:9e:e9:8f:bc:46:bb:dd:25:db:39:57:
81:ed:e5:c8:1f:3d:ca:39:cf:e7:f3:63:75:f6:15:
1f:d4:71:56:ed:84:50:fb:5d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
Netscape Comment:
Puppet Ruby/OpenSSL Internal Certificate
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Subject Key Identifier:
8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
Signature Algorithm: sha1WithRSAEncryption
1d:cd:c6:65:32:42:a5:01:62:46:87:10:da:74:7e:8b:c8:c9:
86:32:9e:c2:2e:c1:fd:00:79:f0:ef:d8:73:dd:7e:1b:1a:3f:
cc:64:da:a3:38:ad:49:4e:c8:4d:e3:09:ba:bc:66:f2:6f:63:
9a:48:19:2d:27:5b:1d:2a:69:bf:4f:f4:e0:67:5e:66:84:30:
e5:85:f4:49:6e:d0:92:ae:66:77:50:cf:45:c0:29:b2:64:87:
12:09:d3:10:4d:91:b6:f3:63:c4:26:b3:fa:94:2b:96:18:1f:
9b:a9:53:74:de:9c:73:a4:3a:8d:bf:fa:9c:c0:42:9d:78:49:
4d:70
root@master:/etc/puppet#
Client's certificate:
client:~# openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/client.example.com.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 3 (0x3)
Signature Algorithm: sha1WithRSAEncryption
Issuer: CN=Puppet CA: master.example.com
Validity
Not Before: Apr 2 20:01:36 2012 GMT
Not After : Apr 2 20:01:36 2017 GMT
Subject: CN=client.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:ae:88:6d:9b:e3:b1:fc:47:07:d6:bf:ea:53:d1:
14:14:9b:35:e6:70:43:e0:58:35:76:ac:c5:9d:86:
02:fd:77:28:fc:93:34:65:9d:dd:0b:ea:21:14:4d:
8a:95:2e:28:c9:a5:8d:a2:2c:0e:1c:a0:4c:fa:03:
e5:aa:d3:97:98:05:59:3c:82:a9:7c:0e:e9:df:fd:
48:81:dc:33:dc:88:e9:09:e4:19:d6:e4:7b:92:33:
31:73:e4:f2:9c:42:75:b2:e1:9f:d9:49:8c:a7:eb:
fa:7d:cb:62:22:90:1c:37:3a:40:95:a7:a0:3b:ad:
8e:12:7c:6e:ad:04:94:ed:47
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:FALSE
Netscape Comment:
Puppet Ruby/OpenSSL Internal Certificate
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Subject Key Identifier:
8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C
X509v3 Extended Key Usage: critical
TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha1WithRSAEncryption
33:1f:ec:3c:91:5a:eb:c6:03:5f:a1:58:60:c3:41:ed:1f:fe:
cb:b2:40:11:63:4d:ba:18:8a:8b:62:ba:ab:61:f5:a0:6c:0e:
8a:20:56:7b:10:a1:f9:1d:51:49:af:70:3a:05:f9:27:4a:25:
d4:e6:88:26:f7:26:e0:20:30:2a:20:1d:c4:d3:26:f1:99:cf:
47:2e:73:90:bd:9c:88:bf:67:9e:dd:7c:0e:3a:86:6b:0b:8d:
39:0f:db:66:c0:b6:20:c3:34:84:0e:d8:3b:fc:1c:a8:6c:6c:
b1:19:76:65:e6:22:3c:bf:ff:1c:74:bb:62:a0:46:02:95:fa:
83:41
client:~#
EDIT:
Someone suggested redoing the certificates. I have done this several times and it does not fix the issue.
My Puppet master contains some sensitive files. I want each puppet agent to be able to access only those files that are of interest to that specific agent. In other words:
- Does the puppet agent run its catalog, and then, whenever it encounters a "file" or "template" function or a "source => 'puppet:///...'" parameter, asks the master to provide it with the specified file, and the master just provides it without checking? This would be bad. If an agent got compromised, it could ask the master for any file on the master, even files which are intended only for other agents.
- Or does the master somehow check that the agent's catalog really authorizes that particular agent to get that particular file?
I don't know if it matters, but I'm running passenger (and all my agents & master are 2.7.6 from squeeze-backports).
Is it possible to setup puppet in a way that changes in manifests only will be applied during certain hours, so that any eventual downtime on our server will occur when we decide it to?
Thanks
When doing a puppet agent call from a new image, I'm getting a err: Could not find class custommod error. The module itself is in /etc/puppet/modules/custommod same as all of the other modules we're calling, but this one is obstinante.
[site.pp]
node /clunod-wk\d+\.sub\.example\.local/ {
include base
include curl
include custommod
class{ "custommod::apps": frontend => "false}
[...]
}
When the puppetmaster is run with debug output, it clearly finding the information for base and curl:
debug: importing '/etc/puppet/modules/base/manifests/init.pp' in environment production
debug: Automatically imported base from base into production
debug: importing '/etc/puppet/modules/curl/manifests/init.pp' in environment production
debug: Automatically imported curl from curl into production
err: Could not find class custommod for clunod-wk0130.sub.example.local at /etc/puppet/manifests/site.pp:84 on node clunod-wk0130.sub.example.local
Line 84 is include custommod
An abbreviated directory and file structure:
/etc/puppet
|- manifests
| |- site.pp
|
|- modules
|- base
| |- manifests
| |- init.pp
|
|- curl
| |- manifests
| |- init.pp
|
|- custommod
|- files
| |- apps
| |- [...]
|
|- manifests
|- init.pp
|- apps.pp
I did check spelling :}
The content of init.pp in the custommod directory is completely unremarkable:
class custommod {
}
The intent is to create an empty class for the apps.pp file, which is where the meat is.
class custommod::apps {
[lots of stuff]
}
Only, it's never getting to the apps file. If I comment out the include custommod, the above error is generated on the class{ "custommod::apps": frontend => "false} line instead.
What am I missing in my hunt to find out how this error is being generated? I need to note that this repo works just fine if it is run locally via puppet apply.