markdrayton's questions

We have a bunch of boxes running memcached. Some are 32-bit and some are 64-bit, but all run a 32-bit install of RHEL due to various Perl-related things. We'll call them type A and B respectively. Both types run memcached 1.2.8 with the same amount of memory allocated via the -m switch. Every memcached instance contains the same data -- in this installation we're not using it as a distributed hash but a local store on each machine. A separate process is responsible for seeding each instance with fresh data at a regular interval.

Today we noticed all the 64-bit machines had started evicting items:

Type A, 32-bit

STAT curr_items 115208
STAT evictions 0
STAT limit_maxbytes 67108864

Type B, 64-bit

STAT curr_items 122192
STAT evictions 1049488
STAT limit_maxbytes 67108864

Initially I thought the type B machines were using more RAM but I was misunderstanding the rusage stats (see edit history). Any ideas why the type B machines would start evicting before the others, even with the same data seeded at the same time as the type A machines? Is it down to the underlying hardware being 64-bit? I'm displaying a shameful ignorance of computer hardware here but there's only one way to fix that.

We have an in-house daemon running on a few RHEL 5 boxes which periodically segfaults. Our developers want a core file to help with debugging but I can't provoke it into producing one.

$ sudo grep segfault /var/log/messages.1
Aug 11 21:04:13 pal108 kernel: brokend[28692]: segfault at 00000000000000a8
rip 00000031d020f908 rsp 00007fff9c60f3f0 error 4

The daemon is started using daemon from /etc/init.d/functions, so adding

DAEMON_COREFILE_LIMIT=unlimited

to its sysconfig file should set ulimit accordingly. This seems to be the case, according to procfs:

$ sudo grep core /proc/$(cat /var/run/brokend.pid)/limits
Max core file size        unlimited            unlimited            bytes  

And the core file pattern points to a location that exists:

$ cat /proc/sys/kernel/core_pattern
"/tmp/core_%p_%e_%t"

Yet it still won't produce a core file. Any ideas what might could be stopping this? Does a segfault always mean the OS will attempt to produce a core file or does it rely on some application-specific coding to do so?

We have a lot of Linux development servers which are generally accessed via SSH. Each developer has a local account on each box, managed by Puppet. Logins are via private keys only; there are no local passwords.

I'd like to run Samba on these boxes and authenticate against our AD domain. I don't want AD authentication for anything besides Samba -- everything else is accessed via SSH and private keys.

Here's my smb.conf:

[global]
 workgroup = DOMAIN
 server string = Samba Server Version %v
 security = ADS
 realm = DOMAIN.FQDN
 encrypt passwords = yes
 log level = 3
 log file = /var/log/samba/%U.log

[homes]
 comment = Home Directories
 browseable = no
 writable = yes

I'm pretty sure the Kerberos configuration is fine as I've joined the domain.

Relevant (ie, non-standard) nsswitch.conf lines:

passwd:     files winbind
group:      files winbind

It looks like the problem is AD UID to UNIX UID mapping. The default TDB backend will create 'virtual' UNIX accounts on demand when AD users connect but I don't want this -- I want user foo to map to the local user foo. If I add idmap uid and idmap gid lines the users authenticate okay but their accounts aren't mapped to the UNIX accounts.

Any ideas? Somoene must've done this before! I don't want to switch to using winbind and AD to provide all the account information because of the hassle with maintaining consistent UID/GIDs on all machines. We've also put a lot into the existing Puppet-controlled user configuration that we don't want to reinvent.

GNU find (and others?) has a -true test along with the normal -name, -mode, -user and so on. From the man page:

-true Always true.

Every time I see the man page I notice this and wonder when it'd be useful. So, give me some examples of when it's useful :~)

We rotate and compress our Apache logs each day but it's become apparent that this isn't frequently enough. An uncompressed log is about 6G, which is getting close to filling our log partition (yep, we'll make it bigger in the future!) as well as taking a lot of time and CPU to compress each day. We have to produce a gziped log for each day for our stats processing. Obviously we could move our logs to a partition with more space but I also want to spread the compression overhead throughout the day.

Using Apache's rotatelogs we can rotate and compress the log more often -- hourly, say -- but how can I concatenate all the hourly compressed logs into a running compressed log for the day, without decompressing the previous logs? I don't want to uncompress 24 hours' worth of data and recompress it because that has all the disadvantages of our current solution.

Gzip doesn't seem to offer any append or concatenate option but perhaps I've missed something obvious. This question suggests straight shell concatenation "works" in that the archive can be decompressed but that gzip -l doesn't work seems a bit dodgy.

Alternatively, perhaps this is still a bad way to do things. Other suggestions are welcome -- our only constraints are our relatively small log partitions and the need to provide a daily compressed log.

We have a number of HP DL385 G2s which kernel panic after installing RHEL 5.3. All are up-to-date with firmware CD 8.50. The initial RHEL 5.3 installation always works and in most cases the first boot is fine (kernel 2.6.18-128.el5); so far 1 in 4 has paniced here. Most of the rest of the machines I've tried fail to boot when "yum update"d to kernel 2.6.18-128.1.10.el5. One or two have been fine.

The panic is always at the same point. The last lines logged on the console are:

device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: [email protected]
usb 4-2: new full speed USB device using uhci_hcd and address 3
device-mapper: dm-raid45: initialized v0.2429
usb 4-2: configuration #1 chosen from 1 choice
hub 4-2:1.0: USB hub found
hub 4-2:1.0: 7 ports detected

then a pause, then:

kernel panic - not syncing - attempted to kill init

Beyond this point no kernel will boot (including the Anaconda-installed 2.6.18-128.el5) and a reinstall is the only option. It seems very similar to this problem reported on the HP forums.

So, any ideas? We have DL385 G2s on RHEL 5.2 so something in 5.3 isn't playing well with the same hardware. We've tried factory resetting the BIOS, etc. How do I work out what the kernel's up to? (I've already removed "rhgb quiet" from the append line.) Luckily we don't have too many of these boxes and I have a little time to investigate.

Should /etc/localtime (under RHEL 5.3, not that I think it matters) be:

1. a symlink to /usr/share/zoneinfo/whatever
2. a hard link to /usr/share/zoneinfo/whatever
3. a copy of /usr/share/zoneinfo/whatever

I'd prefer 1) because it's explicit and easy to manage via Puppet but will it break anything? RedHat's default looks to be 3). Edit: I'm aware of the usual problems of symlinking across filesystems, tzdata updates, etc, but wasn't aware of the historical no-/usr-during-rc.sysinit gotcha. Thanks, all!

We often get DIMMs in our servers going bad with the following errors in syslog:

May  7 09:15:31 nolcgi303 kernel: EDAC k8 MC0: general bus error: participating processor(local node response), time-out(no timeout) memory transaction type(generic read), mem or i/o(mem access), cache level(generic)
May  7 09:15:31 nolcgi303 kernel: MC0: CE page 0xa0, offset 0x40, grain 8, syndrome 0xb50d, row 2, channel 0, label "": k8_edac
May  7 09:15:31 nolcgi303 kernel: MC0: CE - no information available: k8_edac Error Overflow set
May  7 09:15:31 nolcgi303 kernel: EDAC k8 MC0: extended error code: ECC chipkill x4 error

We can use the HP SmartStart CD to determine which DIMM has the error but that requires taking the server out of production. Is there a cunning way to work out which DIMM's bust while the server is up? All our servers are HP hardware running RHEL 5.

How're you analysing log files from UNIX/Linux machines? We run several hundred servers which all generate their own log files, either directly or through syslog. I'm looking for a decent solution to aggregate these and pick out important events. This problem breaks down into 3 components:

1) Message transport

The classic way is to use syslog to log messages to a remote host. This works fine for applications that log into syslog but less useful for apps that write to a local file. Solutions for this might include having the application log into a FIFO connected to a program to send the message using syslog, or by writing something that will grep the local files and send the output to the central syslog host. However, if we go to the trouble of writing tools to get messages into syslog would we be better replacing the whole lot with something like Facebook's Scribe which offers more flexibility and reliability than syslog?

2) Message aggregation

Log entries seem to fall into one of two types: per-host and per-service. Per-host messages are those which occur on one machine; think disk failures or suspicious logins. Per-service messages occur on most or all of the hosts running a service. For instance, we want to know when Apache finds an SSI error but we don't want the same error from 100 machines. In all cases we only want to see one of each type of message: we don't want 10 messages saying the same disk has failed, and we don't want a message each time a broken SSI is hit.

One approach to solving this is to aggregate multiple messages of the same type into one on each host, send the messages to a central server and then aggregate messages of the same kind into one overall event. SER can do this but it's awkward to use. Even after a couple of days of fiddling I had only rudimentary aggregations working and had to constantly look up the logic SER uses to correlate events. It's powerful but tricky stuff: I need something which my colleagues can pick up and use in the shortest possible time. SER rules don't meet that requirement.

3) Generating alerts

How do we tell our admins when something interesting happens? Mail the group inbox? Inject into Nagios?

So, how're you solving this problem? I don't expect an answer on a plate; I can work out the details myself but some high-level discussion on what is surely a common problem would be great. At the moment we're using a mishmash of cron jobs, syslog and who knows what else to find events. This isn't extensible, maintainable or flexible and as such we miss a lot of stuff we shouldn't.

Updated: we're already using Nagios for monitoring which is great for detected down hosts/testing services/etc but less useful for scraping log files. I know there are log plugins for Nagios but I'm interested in something more scalable and hierarchical than per-host alerts.