Banjer's questions

I have the following exec that joins a Linux (CentOS 6) host to an Active Directory domain. When run as root from the bash terminal, it runs successfully and the host is joined to the AD domain properly.

However, when run in puppet, the net ads join command fails with:

Failed to join domain: Failed to set password for machine account (NT_STATUS_ACCESS_DENIED)

Here is the exec

exec { 'adjoin':
    command  => "kinit [email protected] -k -t /etc/krb5.keytab && net ads join createcomputer='Machines/Servers/Linux Servers' osName='${operatingsystem}' osVer=${operatingsystemrelease} -k",
    unless   => "net ads testjoin -k | grep -q 'Join is OK'",
    provider => shell,
    user     => root,
    path     => '/usr/sbin:/usr/bin:/sbin:/bin',
    require  => [
        File['/etc/krb5.conf'],
        File['/etc/krb5.keytab'],
    ],
    logoutput => true,
}

I've tried with and without the provider and user parameters.

When using the net ads join command to join a Linux host (CentOS 6.5) to a domain (Windows Server 2008 DCs), an A record is created but not a corresponding PTR record. Is it possible to have the PTR record created automatically? Perhaps a parameter to net ads join or a setting in /etc/samba/smb.conf?

FYI the full command I run to join a host to the DC:

sudo net ads join createcomputer="machines/Servers/Linux Servers" osName="CentOS" osVer=6.5 -U banjer

When creating a DNS A record manually on the Windows Server the PTR record is created automatically. Let me know if you need any more info.

I've been seeing a bunch of these log messages

Jan 3 00: 58: 57 foo kernel: set_rtc_mmss: can't update from 0 to 58

They occur on CentOS 6.4 VMs running on VMware. I understand it's something to do with the hardware clock not being set properly on the guest OS. I found this command which sets the hardware clock to the current system time:

sudo hwclock --systohc

Is this the correct setting for a virtual machine? Also, where can this be set so it's persistent? In the kernel boot parameters? I'd like for newly provisioned VMs to not have this problem.

UPDATE 1

As requested:

me@foo:~> ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l   43   64  377    0.000    0.000   0.000
+dtc-nist01.ntp. .ACTS.           1 u  174 1024  377    3.311   -8.554   0.497
*nist1-nj.ustimi .ACTS.           1 u  205 1024  377    6.737    3.775   0.433
+nist1-pa.ustimi .ACTS.           1 u   55 1024  377    8.610    4.688   0.337

I see vmwaretools is out of date on this VM. Perhaps the puppet module I have for managing the vmwaretools install did not install it properly. I'll take a look and get back to you.

UPDATE 2

Yes, vmware tools is installed and at the latest version.

me@foo:~> ps aux | grep vmtools
root     56021  0.0  0.1  59508  4156 ?        S    Jan09   3:29 /usr/sbin/vmtoolsd

UPDATE 3

I tried enabling the "Synchronize guest time with host" on the VM:

me@foo:~> vmware-toolbox-cmd timesync status
Disabled
me@foo:~> vmware-toolbox-cmd timesync enable
me@foo:~> vmware-toolbox-cmd timesync status
Enabled

but I'm still getting those messages. In fact, date and hwclock --show are several minutes apart now, whereas they used to be pretty tight.

In the past, older SLES 9 VMs benefited from settings in the VMware article Timekeeping best practices for Linux guests, but it states that CentOS/RHEL 6 guests do not need any additional kernel parameters set.

UPDATE 4

Upgrading to CentOS 6.5 did not help. The kernel is:

Linux foo 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

I've got a NetApp as my nfs server, and two Linux servers as the nfs clients. The problem is that the newer of the two servers has extremely differing read and write speeds whenever it is doing read and writes simultaneously to the nfs server. Separately, reads and writes look great for this new server. The older server does not have this issue.

Old host: Carp

Sun Fire x4150 with w/ 8 cores, 32 GB RAM

SLES 9 SP4

Network driver: e1000

me@carp:~> uname -a
Linux carp 2.6.5-7.308-smp #1 SMP Mon Dec 10 11:36:40 UTC 2007 x86_64 x86_64 x86_64 GNU/Linux

New host: Pepper

HP ProLiant Dl360P Gen8 w/ 8 cores, 64 GB RAM

CentOS 6.3

Network driver: tg3

me@pepper:~> uname -a
Linux pepper 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

I'll jump to some graphs illustrating the read/write tests. Heres pepper and its unbalanced read/write:

and here is carp, lookin' good:

The tests

Here are the read/write tests I am running. I've run these separately and they look great on pepper, but when run together (using the &), the write performance remains solid while the read performance suffers greatly. The test file is twice the size of the RAM (128 GB for pepper, and 64 GB was used for carp).

# write
time dd if=/dev/zero of=/mnt/peppershare/testfile bs=65536 count=2100000 &
# read 
time dd if=/mnt/peppershare/testfile2 of=/dev/null bs=65536 &

The NFS server hostname is nfsc. The Linux clients have a dedicated NIC on a subnet thats separate from anything else (i.e. different subnet than primary IP). Each Linux client mounts an nfs share from server nfsc to /mnt/hostnameshare.

nfsiostat

Heres a 1-minute sample during pepper's simul r/w test:

me@pepper:~> nfsiostat 60

nfsc:/vol/pg003 mounted on /mnt/peppershare:

   op/s         rpc bklog
1742.37            0.00
read:             ops/s            kB/s           kB/op         retrans         avg RTT (ms)    avg exe (ms)
                 49.750         3196.632         64.254        0 (0.0%)           9.304          26.406
write:            ops/s            kB/s           kB/op         retrans         avg RTT (ms)    avg exe (ms)
                1642.933        105628.395       64.293        0 (0.0%)           3.189         86559.380

I don't have nfsiostat on the old host carp yet, but working on it.

/proc/mounts

me@pepper:~> cat /proc/mounts | grep peppershare 
nfsc:/vol/pg003 /mnt/peppershare nfs rw,noatime,nodiratime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.x.x.x,mountvers=3,mountport=4046,mountproto=tcp,local_lock=none,addr=172.x.x.x 0 0

me@carp:~> cat /proc/mounts | grep carpshare 
nfsc:/vol/pg008 /mnt/carpshare nfs rw,v3,rsize=32768,wsize=32768,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,timeo=60000,retrans=3,hard,tcp,lock,addr=nfsc 0 0

Network card settings

me@pepper:~> sudo ethtool eth3
Settings for eth3:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 4
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x000000ff (255)
        Link detected: yes

me@carp:~> sudo ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
        Link detected: yes

Offload settings:

me@pepper:~> sudo ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off

me@carp:~> # sudo ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

Its all on a LAN with a gigabit switch at full duplex between the nfs clients and nfs server. On another note, I see quite a bit more IO wait on the CPU for pepper than carp, as expected since I suspect its waiting on nfs operations.

I've captured packets with Wireshark/Ethereal, but I'm not strong in that area, so not sure what to look for. I don't see a bunch of packets in Wireshark that are highlighted in red/black, so thats about all I looked for :). This poor nfs performance has manifested in our Postgres environments.

Any further thoughts or troubleshooting tips? Let me know if I can provide further information.

UPDATE

Per @ewwhite's comment, I tried two different tuned-adm profiles, but no change.

To the right of my red mark are two more tests. The first hill is with the throughput-performance and the second is with enterprise-storage.

nfsiostat 60 of enterprise-storage profile

nfsc:/vol/pg003 mounted on /mnt/peppershare:

   op/s         rpc bklog
1758.65            0.00
read:             ops/s            kB/s           kB/op         retrans         avg RTT (ms)    avg exe (ms)
                 51.750         3325.140         64.254        0 (0.0%)           8.645          24.816
write:            ops/s            kB/s           kB/op         retrans         avg RTT (ms)    avg exe (ms)
                1655.183        106416.517       64.293        0 (0.0%)           3.141         159500.441

Update 2

sysctl -a for pepper

I'm investigating some poor nfs performance on a new server, as compared to an old server. Both the old and the new server are nfs clients, and they connect to the same exact NetApp nfs server.

My specific question: How to I change the "Advertised pause frame use" setting?

I took a look on the switch side, and the ports on the switch say 1000 Full duplex for both hosts. However, the flow control for the old server shows "Sy/Asy" but the new server is set to Sym. I assume that Asy=Asymmetric and Sym=Symmetric, and my other assumption is that the switch sees Sym because Advertised pause frame use is set to Symmetric on the Linux side. How can I change this to Asymmetric? Could this setting affect performance? I'm just trying to rule out as much as possible.

These are dedicated storage links, i.e. they're a separate NIC and on a separate subnet than the primary NIC.

I see no iowait issues on both the old and new servers, so the CPUs and memory don't appear to be an issue. I figure its something with the eth settings or nfs settings. I can go into more detail on setup, but wanted to answer this specific question first.

New host (CentOS 6.3 on HP ProLiant Dl360P Gen8)

$ sudo ethtool eth3
Settings for eth3:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        **Advertised pause frame use: Symmetric**
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 4
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x000000ff (255)
        Link detected: yes

Driver: tg3

Old host (SLES 9 SP4 on Sun Fire x4150)

$ sudo ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
        Link detected: yes

Driver: e1000

We set up a MongoDB development server on CentOS 6.3 and were able to separate different projects by using separate config files and rc scripts. Now we're looking at setting up a MongoDB production environment.

I've read that it is not recommended to host multiple instances of MongoDB on the same server in production. Does that mean each project will need its own production MongoDB environment?

These projects are not very "big" and so do not require a lot of resources, so it feels like we're jumping the gun by giving each its own host. Perhaps we just need to get our heads out of the RDBMS world.

We will be monitoring our development server to see how it fares, but I'm looking for some insight and some of your own personal experience to supplement what I've read.

I have fail2ban configured on some CentOS 5 and 6 servers, and it sends me an email with a whois of the IP whenever an IP is banned. Is it possible to configure fail2ban to also send a notification to the email from the whois report?

Here is my jail config:

# /etc/fail2ban/jail.conf    

[ssh-iptables]

enabled  = true
filter   = sshd
action   = iptables-allports[name=SSH, protocol=all]
           sendmail-whois[name=SSH, [email protected], sender=fail2ban]
logpath  = /var/log/secure
maxretry = 3

Is there some sort of variable I can put it dest= to send to the whois email?

I've noticed these kerberos keytab error messages on both SLES 11.2 and CentOS 6.3:

sshd[31442]: pam_krb5[31442]: error reading keytab 'FILE: / etc/ krb5. keytab'

/etc/krb5.keytab does not exist on our hosts, and from what I understand of the keytab file, we don't need it. Per this kerberos keytab introduction:

A keytab is a file containing pairs of Kerberos principals and encrypted keys (these are derived from the Kerberos password). You can use this file to log into Kerberos without being prompted for a password. The most common personal use of keytab files is to allow scripts to authenticate to Kerberos without human interaction, or store a password in a plaintext file.

This sounds like something we do not need and is perhaps better security-wise to not have it.

How can I keep this error from popping up in our system logs? Here is my krb5.conf if its useful:

banjer@myhost:~> cat /etc/krb5.conf
# This file managed by Puppet
#
[libdefaults]
        default_tkt_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        default_tgs_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        preferred_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
        default_realm = FOO.EXAMPLE.COM
        dns_lookup_kdc = true
        clockskew = 300

[logging]
        default = SYSLOG:NOTICE:DAEMON
        kdc = FILE:/var/log/kdc.log
        kadmind = FILE:/var/log/kadmind.log

[appdefaults]
pam = {
        ticket_lifetime = 1d
        renew_lifetime = 1d
        forwardable = true
        proxiable = false
        retain_after_close = false
        minimum_uid = 0
        debug = false
        banner = "Enter your current"
}

Let me know if you need to see any other configs. Thanks.

EDIT

This message shows up in /var/log/secure whenever a non-root user logs in via SSH or the console. It seems to only occur with password-based authentication. If I do a key-based ssh to a server, I don't see the error. If I log in with root, I do not see the error. Our Linux servers authenticate against Active Directory, so its a hearty mix of PAM, samba, kerberos, and winbind that is used to authenticate a user.

A postgres SELECT query ran out of control on our DB server and started eating up tons of memory and swap until the server ran out of memory. I found the particular process via ps aux | grep postgres and ran kill -9 pid. This killed the process and the memory freed up as expected. The rest of the system and postgres queries appeared to be unaffected. This server is running postgres 9.1.3 on SLES 9 SP4.

However, one of our developers chewed me out for killing a postgres process with kill -9, saying that it will take down the entire postgres service. In reality, it did not. I've done this before a handful of times and have not seen any negative side effects.

With that said, and after further reading, it looks like kill pid without the flags is the preferred way to kill a runaway postgres process, but per other users in the postgres community, it also sounds like postgres has "gotten better" over the years such that kill -9 on an individual query process/thread is no longer a death sentence.

Can someone enlighten me on the proper way to kill a runaway postgres process as well as the how disastrous (or benign) using kill -9 is with Postgres these days? Thanks for the insight.

I have a physical SLES 11 SP2 server on a Sun Fire x4140 that is giving me problems with networking upon reboot. The NICs are onboard.

The networking appears successful during boot, but network services such as nfs fail hard. This is because eth0 and eth1 are both receiving the same configuration and are both ifup-ed. Once everything times out and I'm at the console, ifconfig shows that eth0 and eth1 are UP and running with the same IP. Attempting to ping anything in that subnet fails. Restarting the network service fixes the issue.

eth0 is the correct NIC that should be configured as primary, per the MAC address.

Question: Whats causing eth1 to be brought up with the same config as eth0??

I do not have a config script set up for eth1:

banjer@harp:~> ls -la /etc/sysconfig/network/
total 104
drwxr-xr-x 6 root root  4096 Jun 11 12:21 .
drwxr-xr-x 6 root root  4096 Apr 10 09:46 ..
-rw-r--r-- 1 root root 13916 Apr 10 09:32 config
-rw-r--r-- 1 root root  9952 Apr 10 09:36 dhcp
-rw------- 1 root root   180 Jun 11 12:21 ifcfg-eth0
-rw------- 1 root root   180 Jun 11 12:21 ifcfg-eth3
-rw------- 1 root root   172 Feb  1 08:32 ifcfg-lo
-rw-r--r-- 1 root root 29333 Feb  1 08:32 ifcfg.template
drwxr-xr-x 2 root root  4096 Apr 10 09:32 if-down.d
-rw-r--r-- 1 root root   239 Feb  1 08:32 ifroute-lo
drwxr-xr-x 2 root root  4096 Apr 10 09:33 if-up.d
drwx------ 2 root root  4096 May  5  2010 providers
-rw-r--r-- 1 root root    25 Nov 16  2010 routes
drwxr-xr-x 2 root root  4096 Apr 10 09:36 scripts

On a side note, eth3 is also configured with an IP in a different subnet, but this has not posed any problems. FYI the kernel module being used is forcedeth.

banjer@harp:~> sudo cat /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='172.21.64.25/20'
MTU=''
NAME='MCP55 Ethernet'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
ONBOOT="yes"

Here's eth3 in case you need to see it:

banjer@harp:~> sudo cat /etc/sysconfig/network/ifcfg-eth3
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='172.11.200.4/24'
MTU=''
NAME='MCP55 Ethernet'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
ONBOOT="yes"

Perhaps is something related to udev? 70-persistent-net-rules looks OK to me, but I may not understand it completely.

banjer@harp:~> cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x10de:0x0373 (forcedeth)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:18:4f:8d:85:4c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x10de:0x0373 (forcedeth)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:18:4f:8d:85:4a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x10de:0x0373 (forcedeth)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:18:4f:8d:85:4b", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x10de:0x0373 (forcedeth)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:18:4f:8d:85:4d", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

# PCI device 0x1077:0x3032 (qla3xxx)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:c1:dd:0e:34:6c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Any other thoughts on what would cause this?

UPDATE 1

Per suggestions, I gave a config to all the other NICs not being used (eth1 and eth2) e.g. here is eth1:

banjer@harp:/etc/sysconfig/network> sudo cat ifcfg-eth1
BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME='MCP55 Ethernet'
NETMASK='255.255.255.0'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='off'
ONBOOT='no'
USERCONTROL='no'

and added the specific HWADDR to the NICs that are actually plugged in (eth0 and eth3). During the test reboot, I see the networking come up as expected, and eth1 and eth2 say "skipped" as expected. However, eth1 is still getting brought up with eth0's config.

I set udev_log="debug" in /etc/udev/udev.conf, and now I have a bunch of debug messages in /var/log/messages. Here is a paste of grep eth1 /var/log/messages, but I don't see anything that stands out when comparing to a grep of other eth's.

UPDATE 2

Thinking this is a udev issue, I made a change to /lib/udev/rules.d/75-persistent-net-generator.rules and did rm /etc/udev/rules.d/70-persistent-net.rules.

# device name whitelist
#KERNEL!="eth*|ath*|wlan*[0-9]|msh*|ra*|sta*|ctc*|lcs*|hsi*", GOTO="persistent_net_generator_end"
KERNEL!="eth[03]|ath*|wlan*[0-9]|msh*|ra*|sta*|ctc*|lcs*|hsi*", GOTO="persistent_net_generator_end"

After rebooting, this did exactly what I wanted (generated rules for eth0, eth3) but it did not solve the problem. eth1 is still brought up. Is there a way to debug the entire boot process, e.g. strace? I have no idea where this is coming from.

As a band-aid, I'm adding an rc script to restart the network late in the boot process.

I recently revoked/cleaned a Puppet agent cert, and this seems to have negative effects in PuppetDB. I see a bug has been filed here with some instructions on fixing the issue. A user had a similar issue here, but none of this is working for me.

The server is running CentOS 6.2, Puppet 2.7.13, and Puppet DB 0.9. The error is:

root@harp:/etc/puppetdb/ssl> puppet agent --test
err: Cached facts for harp failed: Failed to find facts from PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
info: Loading facts in /etc/puppet/modules/dns/lib/facter/datacenter.rb
info: Caching facts for harp
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
err: Could not run Puppet configuration client: Could not retrieve local facts: Failed to submit 'replace facts' command for harp to PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

NTP is working properly from what I see and the datetime looks good. "harp" is actually the puppet master server, so there shouldn't be an issue with time between the agent and server here since they're the same.

Old certificate:

root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp  (DF:8F:65:36:58:4C:DE:66:2B:65:D1:E6:18:B7:F2:33)

Clean and generate new cert for agent:

root@harp:/etc/puppetdb/ssl> puppet cert clean harp
notice: Revoked certificate with serial 18
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/ca/signed/harp.pem'
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/certs/harp.pem'
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/certificate_requests/harp.pem'
notice: Removing file Puppet::SSL::Key harp at '/var/lib/puppet/ssl/private_keys/harp.pem'

root@harp:/etc/puppetdb/ssl> puppet agent --test
info: Creating a new SSL key for harp
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for harp
info: Certificate Request fingerprint (md5): 72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled

root@harp:/etc/puppetdb/ssl> puppet cert list
  harp (72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD)

root@harp:/etc/puppetdb/ssl> puppet cert sign harp
notice: Signed certificate request for harp
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/ca/requests/harp.pem'

root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp  (4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79)

root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb: /etc/init.d/puppetdb: line 77: kill: (8623) - No such process
                                                           [FAILED]
Starting puppetdb:                                         [  OK  ]

OK then, restart again for good measure:

root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb:                                         [  OK  ]
Starting puppetdb:                                         [  OK  ]

Run the SSL configuration script

root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
cp: cannot stat `/var/lib/puppet/ssl/certs/harp.pem': No such file or directory

root@harp:/etc/puppetdb/ssl> ls -la /var/lib/puppet/ssl/certs
total 12
drwxr-xr-x 2 puppet root 4096 Jun 19 07:19 ./
drwxrwx--x 8 puppet root 4096 Apr 24 10:04 ../
-rw-r--r-- 1 puppet root 1854 Apr 24 10:04 ca.pem

OK then, try again for good measure:

root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
Certificate was added to keystore
Usage: pkcs12 [options]
where options are
-export       output PKCS12 file
-chain        add certificate chain
-inkey file   private key if not infile
-certfile f   add all certs in f
-CApath arg   - PEM format directory of CA's
-CAfile arg   - PEM format file of CA's
-name "name"  use name as friendly name
-caname "nm"  use nm as CA friendly name (can be used more than once).
-in  infile   input filename
...snip...
-CSP name     Microsoft CSP name
-LMK          Add local machine keyset attribute to private key

It does not appear that the keystores in /etc/puppetdb/ssl have changed/regenerated. At this point, running puppet agent --test results in the same errors, and restarting puppet and puppetdb do not help.

Keystore info:

root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/keystore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

harp.mydomain.com, May 25, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): 06:A8:D3:2A:70:F3:6D:34:62:91:45:22:8A:C4:A8:86
root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

puppetdb ca, May 25, 2012, trustedCertEntry,
Certificate fingerprint (MD5): 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp.mydomain.com
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
err: Could not call fingerprint: Could not find a certificate or csr for harp.mydomain.com

root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
harp 4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79

How can I get the puppetdb keystore to actually regenerate? I tried deleting the files in /etc/puppetdb/ssl/, but no luck.

I'm using Puppet Dashboard as my ENC and I'm not sure how to reference or use class and group classifications from /etc/puppet/manifests/site.pp.

I have two groups defined in the dashboard: CentOS6 and SLES11. What should my site.pp look like if I want to include a certain list of modules in the CentOS6 group and a certain list of modules in the SLES11 group?

I'm trying to do something like this:

# /etc/puppet/manifests/site.pp

node basenode {
  include hosts
  include ssh::server
  include ssh::client
  include authentication
  include sudo
  include syslog
  include mail
}

node 'CentOS6' inherits basenode {
  include profile
}

node 'SLES11' inherits basenode {
  include usrmounts
}

I have OS-specific case statements within my modules, but there are some modules that will only be applied to a certain distro. So I suppose I have two questions:

1. Is this the best way to apply modules/resources in an OS-specific manner? Or does the above make you want to vomit?
2. Regardless of #1, I'm still curious as how to reference classes, groups, and nodes from Dashboard within my manifests. I've read the External Nodes doc, but I'm not seeing how they correspond to manifests.

Thanks all.

I am trying to configure puppet-dashboard, and I'm running into an issue with Inventory/facts:

Could not retrieve facts from inventory service: 403 "Forbidden request: puppetmasterhostname(ip.address.was.here) access to /facts/agenthostname.example.com [find] at line 99 "

In /etc/puppet/auth.conf on the puppet master:

path /facts
method find
auth any
allow *

I restarted puppetmaster and puppet-dashboard, but I still get the above error. Any ideas or troubleshooting tips?

UPDATE

I am running puppet v2.7.13. As requested, here is my full /etc/puppet/auth.conf. Most of these are defaults that were already in the config:

# allow nodes to retrieve their own catalog (ie their configuration)
path ~ ^/catalog/([^/]+)$
method find
allow $1

# allow nodes to retrieve their own node definition
path ~ ^/node/([^/]+)$
method find
allow $1

# allow all nodes to access the certificates services
path /certificate_revocation_list/ca
method find
allow *

# allow all nodes to store their reports
path /report
method save
allow *

# inconditionnally allow access to all files services
# which means in practice that fileserver.conf will
# still be used
path /file
allow *

### Unauthenticated ACL, for clients for which the current master doesn't
### have a valid certificate; we allow authenticated users, too, because
### there isn't a great harm in letting that request through.

# allow access to the master CA
path /certificate/ca
auth any
method find
allow *

path /certificate/
auth any
method find
allow *

path /certificate_request
auth any
method find, save
allow *

# this one is not stricly necessary, but it has the merit
# to show the default policy which is deny everything else
path /
auth any

# Inventory
path /facts
method find
auth any
allow *

/etc/puppet/puppet.conf

[main]
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet

    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet

    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl

[agent]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig

[master]
   reports = store, http
   reporturl = http://puppetmasterhostname.example.com:3000/reports/upload
   facts_terminus = yaml
   storeconfigs = true
   storeconfigs_backend = puppetdb
   node_terminus = exec
   external_nodes = /usr/bin/env PUPPET_DASHBOARD_URL=http://localhost:3000 /opt/puppet-dashboard/bin/external_node

I'm new to Puppet (open source version) and have a relatively straightforward question.

When I bring up a new host, I'd like the puppetmaster to add the new host's public rsa key to /etc/ssh/ssh_known_hosts, and so the updated ssh_known_hosts file will be available to be pulled down by puppet agents.

I've tried the sshkey resource:

# /etc/puppet/modules/ssh/manifests/client.pp

sshkey { $hostname:
    ensure => present,
    type => "rsa",
    key  => $sshrsakey,
}

However, ssh_known_hosts does not appear to be modified on the puppetmaster, or agent for that matter. My manifest passes syntax validation when I run puppet parser validate client.pp and running puppet agent --test on the agent does not report any issues.

Do I have to have Stored Configs set up in order to use the sshkey resource? I like the features of Stored Configs, but it seems like overkill for what I need and seems to add lots of overhead. My other option is to spit the $sshrsakey fact to a file, but it will need to check for the existence of the public key so it doesn't get added more than once.

I'm configuring CentOS 6.2 and have seen a few "[abrt] full crash report" emails. I understand that abrt is useful for creating crash dumps and what not, so I don't want to disable the service, I just would like to stop getting the crash report emails.

I probably have to add something to the config file in /etc/abrt/abrt.conf. I can't seem to find anything in my searches. Any idea? Thanks.

Edit:

Here is my abrt.conf, which is rather simple.

[root@myhost~]# cat /etc/abrt/abrt.conf
# Enable this if you want abrtd to auto-unpack crashdump tarballs which appear
# in this directory (for example, uploaded via ftp, scp etc).
# Note: you must ensure that whatever directory you specify here exists
# and is writable for abrtd. abrtd will not create it automatically.
#
#WatchCrashdumpArchiveDir = /var/spool/abrt-upload

# Max size for crash storage [MiB] or 0 for unlimited
#
MaxCrashReportsSize = 1000

# Specify where you want to store coredumps and all files which are needed for
# reporting. (default:/var/spool/abrt)
#
#DumpLocation = /var/spool/abrt

And a listing of /etc/abrt:

[root@myhost~]# ls -la /etc/abrt
total 32
drwxr-xr-x.  3 root root  4096 Apr 13 06:14 .
drwxr-xr-x. 97 root root 12288 Apr 13 03:50 ..
-rw-r--r--.  1 root root   527 Dec 13 22:50 abrt-action-save-package-data.conf
-rw-r--r--.  1 root root   572 Dec 13 22:50 abrt.conf
-rw-r--r--.  1 root root   175 Dec 13 22:50 gpg_keys
drwxr-xr-x.  2 root root  4096 Apr 13 06:13 plugins

[root@myhost~]# ls -la /etc/abrt/plugins/
total 12
drwxr-xr-x. 2 root root 4096 Apr 13 06:13 .
drwxr-xr-x. 3 root root 4096 Apr 13 06:14 ..
-rw-r--r--. 1 root root  278 Dec 13 22:50 CCpp.conf

Actually all of those conf files above are only a few lines and do not mention anything about mail, email, or notifications.

UPDATE

Since I have disabled abrtd, I've been seeing these in /var/log/messages:

myhost abrt: abrt daemon is not running. If it crashed, /proc/sys/kernel/core_pattern contains a stale value, consider resetting it to 'core'

The proper way to stop the abrt service is:

service abrt-ccpp stop
chkconfig abrt-ccpp off

I am able to connect to our company's VPN with L2TP on the iphone and ipad, however, I can only get to certain resources in our company network but not others.

After looking at the iOS device logs and routing tables, it seems that only the first 30 routes from our topology are seen on the iOS device. We are running a Checkpoint (R75) firewall.

Is there a limitation on the # of routes that can be saved in a routing table in iOS, or it some kind of limitation of L2TP in general? Any other troubleshooting ideas are welcome. Thanks.

I need to restart snmpd after updating /etc/snmp/conf/snmpd.conf, so it recognizes the changes. I'm using Net-SNMP 5.4.2 on Opensolaris 10. I've tried these two:

snmpd restart
kill -9 pid

The kill command kills it and it fires back up under a new pid, but the new snmpd.conf changes do not seem to be recognized. I'm adding "disk /" to snmpd.conf, and testing to see what filesystems are mounted by:

snmpwalk -v 1 -c public localhost .1.3.6.1.4.1.2021.9.1.2

But this shows nothing, so I'm under the impression my addition of disk / is not taking. Am I restarting snmpd correctly?

I have two machines, each with a 10 Gb adapter, directly connected with an SFP cable. I purchased three SFP cables that look exactly alike (physically and specs), but only one works. I'm having trouble finding clear information about SFP cables, so I appreciate the help in understanding the differences.

Here are the manfacturers, models, etc..

10 Gigabit Card #1: NetXen 3031

10 Gigabit Card #2: QLogic QLE3142-CU-CK

Cable #1 (the one that works) AIPC 7M

Cable #2 (no good) Blade Networks BN-SP-CBL-1M

Cable #3 (no good) Cisco SFP-H10GB-CU3M

I can't even get link lights with the two "no good" cables. I will be ordering a few more of the AIPC cables since they work of course, but I'd like to get a decent understanding of why one works, but not the other. Cable length? Active vs. passive? Different versions/revisions?

I have a warning message that is displayed to a user after they enter their username to log in to our Linux servers. I only want to display this message for password authentications, and not when keys are used. It should only show to a user who is logging into a host, and not when they are sshing from one host to another (our hosts all have keys set up so we can ssh from one to another without entering passwords).

Currently, I have a line for Banner in the /etc/ssh/sshd_config file which points to a text file containing the warning message. Banner doesn't appear to have any further config options, so I'm wondering if there is a way to do this with pam or some other mechanism? Thanks.

I'm setting up an autoinst.xml file for auto-installing SLES 11. I get prompted for the various interface settings per below, but they don't seem to stick once the server reboots. I don't think I have the xml defined correctly. I'm hoping someone has experience with this.

<ask-list>
  <ask>
    <path>networking,dns,hostname</path>
    <question>Enter Hostname (server name)</question>
    <stage>initial</stage>
    <default>merkin</default>
  </ask>
  <ask>
    <path>networking,interfaces,interface,0,device</path>
    <question>Enter the primary ethernet device:</question>
    <stage>initial</stage>
    <default>eth0</default>
  </ask>
  <ask>
    <path>networking,interfaces,interface,0,ipaddr</path>
    <question>Enter the primary IP Address:</question>
    <stage>initial</stage>
    </ask>
  <ask>
    <path>networking,interfaces,interface,0,netmask</path>
    <question>Enter the Netmask Address:</question>
    <stage>initial</stage>
  </ask>
  <ask>
    <path>networking,routing,routes,route,0,gateway</path>
    <question>Enter the primary Gateway Address:</question>
    <stage>initial</stage>
  </ask>
</ask-list>

The first one for hostname seems to be sticking just fine, but the rest do not. As an alternative, is there a way to stop the autoinstall at the section where you configure the network devices so that the user can take over? I was able to show the partition proposal, but not sure how to do the same with the networking setup.