Hamish Downer's questions

I am trying to construct an openssl certificate chain that will work with a standard CentOS 4 install as client. (I know it's very old, but it's what our client is using so we need to support it).

The first problem is that the CentOS 4 openssl CA bundle doesn't contain all the modern certificates, in particular the GoDaddy root certificate

/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority

So I've dug around and found a Valicert certificate for the above certificate and put it in the chain. Running openssl s_client on CentOS 5 (which is openssl 0.9.8e) this chain verifies, but on CentOS 4 (which is openssl 0.9.7a) it doesn't verify.

CentOS 5 output:

$ openssl s_client -CAfile /etc/pki/tls/certs/ca-bundle.crt -connect svn.example.org:443
CONNECTED(00000003)
depth=4 /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
verify return:1
depth=3 /C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
verify return:1
depth=2 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
verify return:1
depth=1 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
verify return:1
depth=0 /OU=Domain Control Validated/CN=*.example.org
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.example.org
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
 2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
   i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
 3 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
   i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
[ SNIP ]
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/CN=*.example.org
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
---
SSL handshake has read 5697 bytes and written 319 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: [ SNIP ]
    Session-ID-ctx:
    Master-Key: [ SNIP ]
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1394712184
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
DONE

While on CentOS 4:

$ openssl s_client -CAfile /usr/share/ssl/certs/ca-bundle.crt -connect svn.example.org:443
CONNECTED(00000003)
depth=4 /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
verify return:1
depth=3 /C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
verify return:1
depth=2 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
verify error:num=7:certificate signature failure
verify return:0
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.example.org
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
 2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
   i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
 3 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
   i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
[ SNIP ]
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/CN=*.example.org
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
---
SSL handshake has read 5697 bytes and written 343 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: [ SNIP ]
    Session-ID-ctx:
    Master-Key: [ SNIP ]
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1394712595
    Timeout   : 300 (sec)
    Verify return code: 7 (certificate signature failure)
---
DONE

Bit flummoxed at this point, any ideas appreciated.

Quick Version

I have set up a private deb repository and copied some signed deb packages into it. I have installed the signing key locally. However when I try to install from the repo I get this warning:

WARNING: The following packages cannot be authenticated!

When installing manually, I can just press y but I want to install these packages automatically using puppet, and that fails.

So what's the problem? Do I need to resign packages with a key I control? Is there a better way of ensuring I have a particular version of puppet installed?

More Details

I have got the packages from the puppet debian repository - http://apt.puppetlabs.com/ I just copy the package from (for lucid) this directory

The repository is then updated with a script that runs these commands for each repo:

cd /var/www/html/apt/ubuntu/lucid
dpkg-scanpackages binary /dev/null | gzip -9c > binary/Packages.gz
dpkg-scansources sources /dev/null | gzip -9c > sources/Sources.gz

I have installed the signing key on the client.

$ sudo apt-key list
/etc/apt/trusted.gpg
--------------------
...
pub   4096R/4BD6EC30 2010-07-10 [expires: 2016-07-08]
uid                  Puppet Labs Release Key (Puppet Labs Release Key) <[email protected]>
...

The rationale for doing this is that I want all puppet clients to be the same version. So all machines should get the packages from my repository by using the following pin in /etc/apt/preferences.d/puppet:

Package: puppet puppet-common facter
Pin: origin deb.example.org
Pin-Priority: 1001

(We are currently using puppet 2.6.x, so I need a priority of 1001 to downgrade precise clients from 2.7.x).

I have read about holding packages but that doesn't help me change package versions.

All suggestions welcome.

I read this bug report about RapidSSL certificates not being recognised by older versions of Android. I've got the necessary certificate chain from here, but I don't have an Android device to test on.

So how can I test if the certificate is set up properly?

I have set up a server with a SSL certificate, added the necessary chain, and done the apache set up. I have tested using:

openssl s_client -CAPath /etc/ssl/certs -connect www.example.org:443

and get various output including:

Verify return code: 0 (ok)

Both firefox and chrome are happy with the site, but some clients (including svn) report the certificate is rejected. What's wrong?

Here is the full output from openssl:

$ openssl s_client -CApath /etc/ssl/certs -connect www.aptivate.org:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = [email protected]
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1
depth=1 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certificates.godaddy.com/repository, CN = Go Daddy Secure Certification Authority, serialNumber = 07969287
verify return:1
depth=0 O = *.aptivate.org, OU = Domain Control Validated, CN = *.aptivate.org
verify return:1
---
Certificate chain
 0 s:/O=*.aptivate.org/OU=Domain Control Validated/CN=*.aptivate.org
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
 1 s:/O=*.aptivate.org/OU=Domain Control Validated/CN=*.aptivate.org
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
 2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
   i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
 3 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2 Certification Authority
   i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
 4 s:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
   i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class 2 Policy Validation Authority/CN=http://www.valicert.com//[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFVzCCBD+gAwIBAgIHJ+uaBU9wOjANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAY
BgNVBAoTEUdvRGFkZHkuY29tLCBJbmMuMTMwMQYDVQQLEypodHRwOi8vY2VydGlm
aWNhdGVzLmdvZGFkZHkuY29tL3JlcG9zaXRvcnkxMDAuBgNVBAMTJ0dvIERhZGR5
IFNlY3VyZSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTERMA8GA1UEBRMIMDc5Njky
ODcwHhcNMTEwMTEyMTcwNjE4WhcNMTQwMTEyMTcwNjE4WjBVMRcwFQYDVQQKDA4q
LmFwdGl2YXRlLm9yZzEhMB8GA1UECwwYRG9tYWluIENvbnRyb2wgVmFsaWRhdGVk
MRcwFQYDVQQDDA4qLmFwdGl2YXRlLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBANogFQnLwi8hKKJ6rm117cJSxeIOmt9dZT+5MsxQO7YesHM/hOuy
/Dk6WI6eaFPgms+w1EgEalbyaagMpnAoCnfQPfXFM221MzKXbD9w0+iXi3nwdCBw
vnsgPsCtaz1uDBrEALN+NympRyb8jGVWCaGY2ypPxpqvO8olkNSvrVhsFy+EdfmA
IPvCinuvBQXaUI6gcoEAM/yFE4u2a0NkXip/vmUgAH8u5Hz/ksx36J+oWpl59ofk
6K+VuudjRU8vpaoYY2HSL01p1w1b6QFHU2RvD/FrQg0spZXflfULULEbjcH69gEL
slg9oqL3coVS8XVN0wwWM10SHUQN4s2rhTsCAwEAAaOCAbQwggGwMA8GA1UdEwEB
/wQFMAMBAQAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA4GA1UdDwEB
/wQEAwIFoDAzBgNVHR8ELDAqMCigJqAkhiJodHRwOi8vY3JsLmdvZGFkZHkuY29t
L2dkczEtMjguY3JsME0GA1UdIARGMEQwQgYLYIZIAYb9bQEHFwEwMzAxBggrBgEF
BQcCARYlaHR0cHM6Ly9jZXJ0cy5nb2RhZGR5LmNvbS9yZXBvc2l0b3J5LzCBgAYI
KwYBBQUHAQEEdDByMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5nb2RhZGR5LmNv
bS8wSgYIKwYBBQUHMAKGPmh0dHA6Ly9jZXJ0aWZpY2F0ZXMuZ29kYWRkeS5jb20v
cmVwb3NpdG9yeS9nZF9pbnRlcm1lZGlhdGUuY3J0MB8GA1UdIwQYMBaAFP2sYTKT
bEXW4u6FX5q653aZaMznMCcGA1UdEQQgMB6CDiouYXB0aXZhdGUub3JnggxhcHRp
dmF0ZS5vcmcwHQYDVR0OBBYEFIfBDt6TU0GGA1uyaEeC3yePHYgoMA0GCSqGSIb3
DQEBBQUAA4IBAQAjJ27yI6vf9QwFIWDiqyTx7jsWYJAGqPG0KsyNgMYBaXdSSY+b
q0dKSRtBwX8sM4sR1ahSNLFlajIy+3dzm/qsk16vKgV+NPgz+Zl7pqm6NaObW+uV
XWw9QHln7jvQXadhV9JXU4/mjF76/0eWs1egLhBn8vgtcqAS4ZckwxLXNmEGm729
IGIppZwe/yy66KJ2tahkCeWBcrNkdcnxsWsD60hfRR4fe8FrQMUxeeIca7c3LguG
0KhtI/fJN+iVORdfG8enJkjZzQW67z/W3gFM1AVIrnmu94DQ9Ro/vD7vp55YkBld
28o9agLg5Dl9cu5Xs7onqcTTJJpCp3AQHQE3
-----END CERTIFICATE-----
subject=/O=*.aptivate.org/OU=Domain Control Validated/CN=*.aptivate.org
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
---
No client certificate CA names sent
---
SSL handshake has read 5325 bytes and written 416 bytes
---
New, TLSv1/SSLv3, Cipher is EDH-RSA-DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : EDH-RSA-DES-CBC3-SHA
    Session-ID: 5098DC94E1FF98BBD0DBA424A973728346F974FF02700928ECA32E27E10992F5
    Session-ID-ctx: 
    Master-Key: DBE2733FE83E8B3105FD1F63D023AF4DFC5BBA028CC1DD35107FDC9F913A88E2F58C65FBC5839525BF4D529A7DBBA91E
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1352195224
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

I'm preparing for a load test of a classroom wifi system. The pupils all turn on their laptops at the start of the lesson, which launches the web browser and then they start the lesson - which involves downloading a flash based lesson (from a server within the school), generally half to 2 MB download.

In some cases, the load time is stretching to 5 or 10 minutes. So I want to monitor all parts of the system to say with confidence where the bottlenecks are, and how many clients can use a single wifi access point reasonably. So we are planning to run tests with up to 50 clients and see what happens (I know that most people recommend 20-25 clients per access point, but the client wants to test this - and I want to get good data to show the client one way or another).

I already know how to monitor the server. The wifi access point supports SNMP, and seems to provide quite a lot of variables, but I don't want to have to wade through too much.

So the question is, what wifi-related variables are the key ones to monitor to characterise when the system is getting overloaded, clients are waiting, collisions are happening etc?

I'm happy to be told the generic names for what should be there and hunt through the files myself, but if you want/need to see the details, then the access point we are using is the Ubiquity Nanostation 2. The MIB files for Ubiquity products are linked from the bottom of their SNMP page. Though I also found that they seem to support at least part of the Mikrotik MIB.

We have our own apt/deb repository with a handful of packages where we want to control the version. Crucially this includes puppet, which can be sensitive to versions being different.

I want our desktops to only get puppet from our repository, but also for people to be able to add their own PPAs, enable backports etc. The current problem we have is backports on Ubuntu Lucid. Some important lines from /etc/apt/sources.list:

deb http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
deb http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
deb http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu/ lucid-security main restricted universe multiverse

deb http://deb.example.org/apt/ubuntu/lucid/ binary/

And in /etc/apt/preferences.d/puppet:

Package: puppet puppet-common
Pin: release a=binary
Pin-Priority: 800

Package: puppet puppet-common
Pin: release a=lucid-backports
Pin-Priority: -10

Currently policy says:

$ sudo apt-cache policy puppet
puppet:
  Installed: (none)
  Candidate: (none)
  Package pin: 2.7.1-1ubuntu3.6~lucid1
  Version table:
     2.7.1-1ubuntu3.6~lucid1 -10
        500 http://gb.archive.ubuntu.com/ubuntu/ lucid-backports/main Packages
        100 /var/lib/dpkg/status
     2.6.14-1puppetlabs1 -10
        500 http://deb.example.org/apt/ubuntu/lucid/ binary/ Packages
     0.25.4-2ubuntu6.8 -10
        500 http://gb.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        500 http://security.ubuntu.com/ubuntu/ lucid-security/main Packages
     0.25.4-2ubuntu6 -10
        500 http://gb.archive.ubuntu.com/ubuntu/ lucid/main Packages

If I use n= instead of a= then I get Package pin: (not found)

I'm just plain confused at this point as to what I should use. Any help appreciated.

Edit

I've spent some more time reading the apt/preferences man page and found the origin keyword. So I've now tried:

Explanation: get puppet from our server
Explanation: priority over 1000 means downgrade if necessary
Package: puppet puppet-common
Pin: origin "deb.example.org"
Pin-Priority: 1001

I did delete the backports line. The man page says that a priority over 1000 will lead to a downgrade. But as it is there is nothing installed. I still get Package pin: (not found) and Candidate: (not found) while still getting the full version table above. I don't see why my origin doesn't change the priority of the deb.example.org one.

I run a number of CentOS servers. I was recently doing the occasional yum update but this failed on a handful of the servers. On most I have been able to move forward by doing:

# yum clean all
# rm -f /var/lib/rpm/__db*
# rpm --rebuilddb
# yum update

However on one server, the yum clean command hangs. I have tried running:

# yum -v --noplugins clean all

but I get no output at all. Yum just hangs, and will sit there for hours if I let it, not using any cpu, just stopped. Doing

# strace -f yum -v --noplugins clean all

produces quite a lot of output, but then stops with:

...
stat64("/var/lib/rpm/__db.003", {st_mode=S_IFREG|0644, st_size=450560, ...}) = 0
open("/var/lib/rpm/__db.003", O_RDWR|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
mmap2(NULL, 450560, PROT_READ|PROT_WRITE, MAP_SHARED, 4, 0) = 0xb6b34000
close(4)                                = 0
stat64("/var/lib/rpm/Packages", {st_mode=S_IFREG|0644, st_size=14938112, ...}) = 0
open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
read(4, "\0\0\0\0\1\0\0\0\0\0\0\0a\25\6\0\10\0\0\0\0\20\0\0\0\10\0\0k\t\0\0"..., 512) = 5
12
close(4)                                = 0
open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=14938112, ...}) = 0
futex(0xb6b7bd1c, FUTEX_WAIT, 1, NULL

At this point I'm feeling stuck. This is a production server, so I don't want to just blow it away, or even blow away all the yum information. Apart from the yum stuff, the server is working fine.

Is there a canonical way to find out the last time that yum update was run on a system?

Our set up is that we have staging servers that run automatic updates, and provided they don't fall over, we will manually update our production servers about once a month (barring critical updates). (I say manually, ideally I want to manually trigger an update across all of them, but that's another issue).

But you get busy, tasks slip etc. So I want to set up a nagios check that will start bothering us if we've left it too long.

Searching the web hasn't got me very far. Poking around the system, the best thing I've found so far would be something like:

grep Updated /var/log/yum.log | tail -1 | cut -d' ' -f 1-2

which gives me something like Mar 12 which I can then convert into a date. There are a few minor complications about whether the date is this year or last year, and I'd also need to check /var/log/yum.log.1 in case of checking immediately after a logrotate. But that is just scripting details.

This can of course be 'fooled' by an update to a single package rather than a general update.

So is there a more canonical way to see when yum update was run?

Edit: I've now written a Nagios NRPE plugin that uses the idea I put forward in the question. You can grab it from https://github.com/aptivate/check_yum_last_update

We have a working LDAP set up. However we hit a problem recently due to some user names and group names existing both in the local files and in LDAP. Specifically the apache user and group existed both in:

• /etc/passwd and /etc/group
• LDAP as both user and group

After a recent yum update (CentOS 5), it appeared that the group ID of the process had changed from the value in /etc/group to the value in LDAP (while the user ID remained the ID from /etc/passwd). As some of the files needed by httpd were owned by user root, with group apache (from /etc/group) but not world readable, this caused problems.

Note that we already have nss_initgroups_ignoreusers apache,... in both /etc/ldap.conf and /etc/openldap/ldap.conf. Also, in /etc/nsswitch.conf we have

passwd:     files ldap
group:     files ldap
shadow:     files ldap

and the rest of the normal bits and bobs.

So if a name exists in both LDAP and the local files, is there a way to ensure that the ID from the local files will take precedence over the ID in LDAP?

When I do a yum check-update on a CentOS machine, I see that some updates are from the base repo, and some from the updates repo. Can anyone tell me what the difference between the two are? I have found the repositories guide but that is only about non-standard repositories - it has nothing about the default repositories.

My best guess would be that the base repository would get updated when CentOS goes from (say) 5.5 to 5.6, while in between any new packages would go into the updates repository. But I can't find anything on the internet about it - the search terms are too general to zero in on it.

As part of my job I manage a few tens of CentOS 5 servers, using puppet for the main set up. About half of our servers have a standardised set up for hosting various django sites, while the rest are a mish mash of applications.

I'm gradually sorting out our hosting practices, and I've now got to the point of working out how to manage security updates at the OS level. I'm wary of just having a cron job doing a yum -y update but also don't want to have to go round each server in time and review every package with updates available, as that would take a while.

So I'm wondering if there are any good shortcuts or working practices that would minimise the risks involved and minimise the amount of time I need to spend. Or to put it another way are there any tools or practices that can automate a lot of the work while still giving control.

Steps I've decided on so far:

• disable all third party repositories and set up our own repository so I can control what updates go through there.
• we have staging servers for (most of) our production servers where I could do testing (but how much testing is enough testing?)

Also note that I've looked into the yum security plugin but it does not work on CentOS.

So how do you manage updates for significant numbers of CentOS servers running a heterogeneous array of applications?

We have a puppet set up using the nagios types. Mostly it works fine, but sometimes I find that the generated puppet-hosts.cfg file keeps growing - the hosts are just added and re-added ad nauseam. Deleting the file and running puppetd -t regenerates it fine.

Our nagios host bit in the puppet nagios module is:

# set up alias
$real_nagios_alias = $nagios_alias ? { '' => $hostname, default => $nagios_alias }
$real_nagios_contact_groups = $nagios_contact_groups ? 
  { '' => 'admins', default => $nagios_contact_groups }
$real_nagios_parents = $nagios_parents ? { '' => '', default => $nagios_parents }

$default_nagios_hostgroups = 'all-servers'
$real_nagios_hostgroups = $nagios_hostgroups ? 
  { '' => $default_nagios_hostgroups, default => $nagios_hostgroups }


@@nagios_host { $hostname:
  ensure          => present,
  address         => $fqdn,
  alias           => $real_nagios_alias,
  contact_groups  => $real_nagios_contact_groups,
  hostgroups      => $real_nagios_hostgroups,
  parents         => $real_nagios_parents,
  use             => 'generic-host',
  target          => $nagios_puppet_host_file,
}

($nagios_puppet_host_file is defined in the main manifests/site.pp file). In the nagios server class we have the line:

Nagios_host              <<||>> { notify => Service['nagios'] }

When running puppetd -t on the nagios server, I often get an error along the lines of:

err: Could not prefetch nagios_host provider 'naginator': Could not parse configuration for nagios_host: line 15: syntax error at '
' in /usr/local/nagios/etc/puppet-hosts.cfg

The line number varies, but when I look in the file, the line number will line up with something like:

define host {
  address                        somehost.example.org
  contact_groups                 admins

The line number will be the line with "address" on it. Apart from that I've not spotted a pattern to which host it complains about.

So is there any reason why the hosts keep getting regenerated?

Puppet version is 0.25.4 generally, including on the puppet server, though it is 2.6.2 on the nagios server, which I guess may be the problem.

I've just set up puppet to generate nagios config files, which is great :) I have the services being put into a file named as $nagios_puppet_service_file and here is part of the nagios service class:

  file { $nagios_puppet_service_file:
    ensure => file,
    owner  => nagios, group => admins, mode => '0664',
    notify => Service['nagios'],
  }

  Nagios_host <<||>>
  Nagios_hostextinfo <<||>>
  Nagios_service <<||>>

  service { 'nagios':
    ensure     => running,
    enable     => true,
    subscribe  => [ File[$nagios_puppet_host_file], File[$nagios_puppet_hostextinfo_file],
                    File[$nagios_puppet_service_file], ],
    hasrestart => true,
    hasstatus  => true,
  }

While one of my service definitions looks like:

$real_nagios_http_port = $nagios_http_port ? { '' => '80',  default => $nagios_http_port }
$real_nagios_http_url  = $nagios_http_url  ? { '' => '/',   default => $nagios_http_url  }
$real_nagios_http_addr = $nagios_http_addr ? { '' => $fqdn, default => $nagios_http_addr }


@@nagios_service { "check_http_port_${hostname}":
  use                 => 'generic-service',
  check_command       => "check_http_port!$real_nagios_http_addr!$real_nagios_http_port!$real_nagios_http_url",  
  service_description => 'HTTP service',
  host_name           => $hostname,
  target              => $nagios_puppet_service_file,
}

So the service file is generated correctly, but the nagios service is not restarted, despite the subscribe clause in the service part above. (Well, sometimes it is restarted when the permissions on the file are updated, but that doesn't always happen on a puppet run, and sometimes happens part way through a puppet run, so only some changes get loaded, despite them all being in the services file.)

Any ideas why the subscribe clause isn't working as expected? Can I subscribe to all nagios_service instances some how? Would a notify in the nagios_service bit work any better?

I thought I'd worked this out, but after reading Continuous Delivery (excellent book) I'm a little confused. They talk about having servers for:

• development
• various forms of automated tests
• user acceptance testing (UAT) - ie sitting down with the client and demonstrating it to them, and letting them do exploratory testing. The in-house testers could also use this setup for exploratory testing.
• staging
• production.

I'd always thought of staging as providing the UAT function, but they seem to have staging as a separate level. So in that scheme, what function would the staging servers provide?

I don't seem to be able to access port 25 on an instance on Amazon EC2. I have opened the port in the security group and in the iptables firewall. I have opened other ports (including 22 and 80) the same way and they work.

On the instance, I can telnet to port 25 and see the output from postfix. From outside EC2 I can telnet to port 22 or port 80 and see the output from the SSH server and the web server. But when I telnet to port 25 from outside EC2 it just sits waiting.

I have filled in the form to be able to send email and had confirmation that that has been done. And I've checked and outgoing email works fine. (All be it through a relay to avoid IP blacklists).

So does Amazon treat incoming traffic to port 25 differently? Does it work fine for other people? Any help appreciated.

As an alternative source of trouble I am using postfix on SUSE Linux (SLES 10) and I am wondering if either of those have some hidden settings that mean they won't respond to port 25 from off the machine. I have checked the iptables settings with iptables -L -nvv and iptables -t nat -L -nvv and I can't see any rules that would get in the way. And postfix does respond to telnet on the localhost.

Edit: When I turn iptables off completely, telnet returns immediately with

telnet: Unable to connect to remote host: Connection refused

As requested by an answer, the contents of master.cf are:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       n       -       -       smtpd
#submission inet n      -       n       -       -       smtpd
#       -o smtpd_etrn_restrictions=reject
#       -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#smtps    inet  n       -       n       -       -       smtpd
#  -o smtpd_tls_wrappermode=yes -o smtpd_sasl_auth_enable=yes
#submission   inet    n       -       n       -       -       smtpd
#  -o smtpd_etrn_restrictions=reject
#  -o smtpd_enforce_tls=yes -o smtpd_sasl_auth_enable=yes
#628      inet  n       -       n       -       -       qmqpd
pickup    fifo  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       n       300     1       oqmgr
#tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
smtp      unix  -       -       n       -       -       smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       n       -       -       smtp
        -o fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
#localhost:10025 inet   n       -       n       -       -       smtpd -o content_filter=
scache    unix  -       -       n       -       1       scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
cyrus     unix  -       n       n       -       -       pipe
  user=cyrus argv=/usr/lib/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=foo argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
procmail  unix  -       n       n       -       -       pipe
  flags=R user=nobody argv=/usr/bin/procmail -t -m /etc/procmailrc ${sender} {recipient}

and grep -v '#' /etc/sysconfig/postfix:

POSTFIX_RELAYHOST="[mail.domain.com]"
POSTFIX_MASQUERADE_DOMAIN=""
POSTFIX_LOCALDOMAINS="\$myhostname"
POSTFIX_NULLCLIENT="no"
POSTFIX_DIALUP="no"
POSTFIX_NODNS="no"
POSTFIX_CHROOT="no"
POSTFIX_UPDATE_CHROOT_JAIL=no
POSTFIX_LAPTOP=no
POSTFIX_UPDATE_MAPS=yes
POSTFIX_MAP_LIST="virtual transport access canonical sender_canonical relocated sasl_passwd:600 relay_ccerts"
POSTFIX_RBL_HOSTS=""
POSTFIX_BASIC_SPAM_PREVENTION=off
POSTFIX_MDA=local
POSTFIX_SMTP_AUTH_SERVER=no
POSTFIX_SMTP_AUTH=no
POSTFIX_SMTP_AUTH_OPTIONS=""
POSTFIX_SMTP_TLS_SERVER=no
POSTFIX_SMTP_TLS_CLIENT="no"
POSTFIX_SSL_PATH="/etc/postfix/ssl"
POSTFIX_TLS_CAFILE="cacert.pem"
POSTFIX_TLS_CERTFILE="certs/postfixcert.pem"
POSTFIX_TLS_KEYFILE="certs/postfixkey.pem"
POSTFIX_SSL_COUNTRY="XX"
POSTFIX_SSL_STATE="Some state"
POSTFIX_SSL_LOCALITY="Some locality"
POSTFIX_SSL_ORGANIZATION="Some Organization"
POSTFIX_SSL_ORGANIZATIONAL_UNIT="Some Organizational Unit"
POSTFIX_SSL_COMMON_NAME="A common name"
POSTFIX_SSL_EMAIL_ADDRESS="postmaster"
POSTFIX_ADD_MAILBOX_SIZE_LIMIT=0
POSTFIX_ADD_MESSAGE_SIZE_LIMIT=10240000
POSTFIX_REGISTER_SLP="yes"
POSTFIX_ADD_MYNETWORKS_STYLE="subnet"

I've changed the domain at the top, but it is otherwise the same.

Possible Duplicate:
Can you help me with my software licensing question?

We have an existing SUSE Linux server with license we want to move to Amazon EC2. I want to know if we can pay the standard Linux prices if using our own license, or if we have to pay the more expensive SUSE price schedule.

The server will be always on (running a website) so we will be using a reserved instance. However I don't want to buy a reserved Linux instance and then find that the SUSE ami doesn't work with that and will only run in a reserved SUSE instance. So does anyone know if the SUSE ami will run in a reserved Linux instance?

The ami we'd like to use is ami-eea35787 (provided by SUSE).

We have some servers on Amazon EC2, and we back up the entire file system using cumulus. I'm wondering how to restore the entire file system from cumulus on to an EC2 instance - we need to do a test restore. Some ways I can imagine:

• Amazon has some way of filling the root filesystem before boot.
• Set up an instance and restore the files to /tmp. To copy it all to root I'd have to stop the instance running, otherwise there would be problems with open files, but I'm not sure how to access the filesystem of a stopped instance.

And, yes, I am new to EC2, so I may be missing some obvious tools.

I need two different versions of ruby on a server. The packaged ruby for the packaged puppet to work, and a compiled ruby for a rails web app to work. So basically I want the default ruby to be the ruby from the package, and for rails to use the compiled ruby (in /usr/local/bin/ and /usr/local/lib/ )

I've found references to setting the RUBYPATH and RUBYLIB environment variables, but I'm confused as to where to set them. In .bashrc, .profile, in the apache config somewhere?

I am running apache2 on Debian etch, with multiple virtual hosts.

I want to redirect so that http://git.example.com goes to http://git.example.com/git/

Should be really simple, but Google isn't quite cutting it. I've tried the Redirect and Rewrite stuff and they don't quite seem to do what I want ...

My organisation has been looking into how to spread our servers around geographically while keeping backups very up to date, and ideally spreading the load.

The initial thing I have in mind is Rails on MySQL. The write rate isn't too high (articles/comments being left at less than 1 per minute, though some have large media attachments).

So,

• does MySQL replication work well across wide area networks?
• Does the connection (or a slave server) going down mean that manual intervention is required (once the two servers can talk to each other again) or is recovery automatic?
• If the master disappears, what is required to turn a slave into a master? Are there standard scripts/tools to help manage that?
• Any other gotchas etc?