Insyte's questions

I'm running MySQL on an Ubuntu 10.04 server, with the MySQL $datadir located on an iSCSI volume. The filesystem is tagged with "_netdev" in /etc/fstab. On system shutdown, upstart sends MySQL a SIGTERM, then moves on to shutting down the network interfaces, without waiting for MySQL to exit or for the iSCSI disk to be unmounted or disconnected. This, of course, results in XFS not unmounting cleanly and MySQL needing to spend several minutes replaying InnoDB logs on startup.

I'm not clear on the right place to begin poking to fix this. From what I've read, upstart should already be smart enough to wait for iSCSI filesystems to unmount and disconnect before it shuts down the network interfaces, although this isn't happening. And I have no idea how to tell it that it also needs to wait for a clean shutdown of MySQL.

This is what the console shows during the shutdown process:

Broadcast message from root@mysql-a1
    (unknown) at 11:19 ...

The system is going down for halt NOW!
Power button pressed
 * Stopping Bacula File daemon...                                        [ OK ]
Stopping file integrity checker: samhain.
 * Running nssldap-update-ignoreusers...                                 [ OK ]
 * Stopping multipath daemon multipathd                                  [ OK ]
 * Stopping nagios-nrpe nagios-nrpe                                      [ OK ]
 * Stopping Name Service Cache Daemon nscd                               [ OK ]
 * Stopping Postfix Mail Transport Agent postfix                         [ OK ]
 * Stopping SSH throttling throttle-ssh                                  [ OK ]
 * Stopping puppet agent                                                 [ OK ]
 * Stopping system logging syslog-ng                                     [ OK ]
Stopping statistics collection and monitoring daemon: collectdcollectd[1210]: Exiting normally.
collectd[1210]: collectd: Stopping 15 read threads.
collectdmon[1209]: Info: collectd terminated with exit status 0
collectdmon[1209]: Info: shutting down collectdmon
.
 * Asking all remaining processes to terminate...                        [ OK ]
 * All processes ended within 1 seconds....                              [ OK ]
 * Deconfiguring network interfaces...                                          [ 2884.248199] end_request: I/O error, dev dm-0, sector 5216
[ 2884.249807] end_request: I/O error, dev dm-0, sector 4192
[ 2884.817855] end_request: I/O error, dev dm-0, sector 0
[ 2884.819347] XFS (dm-0): Device dm-0: metadata write error block 0x0
[ 2884.821281] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[ 2884.823393] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[ 2885.367423] end_request: I/O error, dev dm-0, sector 0
init: mysql main process (4168) terminated with status 1
                                                                         [ OK ]
 * Deactivating swap...                                                  [ OK ]
 * Unmounting weak filesystems...                                        [ OK ]
 * Unmounting local filesystems...                                       [ OK ]
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Will now halt
[ 2886.802324] Power down.
Connection to bottom.cw closed.
bottom:~ insyte$

This is the fstab entry:

/dev/mapper/21db3d79bf30ef4846c9ce90069680087 /srv/mysql xfs  _netdev,noatime,nodev,noexec  0 0

Previous PERC controllers could all be managed/monitored by the LSI MegaCLI tools. The latest release of those tools does not appear to recognize my H700 card:

$ sudo /opt/MegaRAID/MegaCli/MegaCli64 -adpCount 

Controller Count: 0.

Are there any options aside from installing the massive OMSA suite?

-Ben

I use a redundant pair of OpenLDAP servers for PAM auth and directory services via NSS. It's been 100% reliable so far, but nothing runs flawlessly forever.

What steps should I take now so I have a fighting chance of recovering from failure of the LDAP server(s)? In my informal testing, it appears that even already authenticated shells are largely useless as all username/uid lookups hang until the directory server comes back.

So far I've come up with only two things:

1. Do not use NSS-LDAP and PAM-LDAP on the LDAP servers themselves.
2. Create a root-level account on all boxes that only accepts publickey authentication from our local subnet and protect that key well. I'm not sure how much good this would do me as once I'm logged in, I suspect I wouldn't be able to accomplish anything since all the userid lookups would be hanging.

Any other suggestions?

I'm trying to set up a couple of simple rules that rewrite URLs from an old version of a site based on bits of the path and the query string.

First whack at it looked like this:

RewriteEngine on

RewriteMap freeitems txt:/www/www.example.com/etc/freeitems_map.txt
RewriteMap items txt:/www/www.example.com/etc/items_map.txt

RewriteCond %{QUERY_STRING} ^sku=(\d+)$
RewriteRule ^/free_item.php$ https://www.example.com/shop/detail/${freeitems:%1}?
RewriteRule ^/item.php$ https://www.example.com/shop/detail/${items:%1}?

That seemed to work great for the /free_item.php URLs, properly subsituting in the result from the RewriteMap, but /item.php URLs returned just the first part of the redirected URL, leaving out the results from the RewriteMap lookup. If I reversed the order of the rules, the failed/working URLs reversed. As I have always understood RewriteRule syntax, if the URL path doesn't match the pattern, the rule is skipped and processing moves on to the next one. In this case, the

I finally managed to get it working by breaking it out into two blocks:

RewriteEngine on

RewriteMap freeitems txt:/www/www.example.com/etc/freeitems_map.txt
RewriteMap items txt:/www/www.example.com/etc/items_map.txt

RewriteCond %{QUERY_STRING} ^sku=(\d+)$
RewriteRule ^/free_item.php$ https://www.example.com/shop/detail/${freeitems:%1}?

RewriteCond %{QUERY_STRING} ^sku=(\d+)$
RewriteRule ^/item.php$ https://www.example.com/shop/detail/${items:%1}?

Can anyone enlighten me as to why the first technique failed?

EDIT: As a test, I added an additional path element to the redirected URL for the second RewriteRule. When I requested a URL that should match the second rule and not the first, that path element appears as expected. Just not the result of the RewriteMap lookup.

Thanks!

-Ben

I'm trying to create a one-off snapshot of 5.5TB of data to 3 external 2TB drives. The data is on an XFS partition, so the logical option seemed to be xfsdump, as it can span multiple devices. As a test, I've created a number of smaller partitions on one of the drives so I can force it to reach the end of the drive in minutes instead of hours. The external drive contains two partitions, /dev/sde1 and /dev/sde2, both 256MB in size.

This is the command I'm using:

xfsdump -o -l0 -s daily.0/jones -f /dev/sde2 -p 10 /snaps

I'm prompted for dump and media labels and the backup begins. After around 30 seconds, I get these messages:

xfsdump: ending media file
xfsdump: media file size 750000128 bytes
xfsdump: dump size (non-dir files) : 728189552 bytes
xfsdump: NOTE: dump interrupted: 27 seconds elapsed: may resume later using -R option
xfsdump: Dump Status: INTERRUPT

That's pretty much what I was hoping to see, as it hit the end of the available space on /dev/sde2.

Running xfsdump -I looks good, too:

# xfsdump -I
file system 0:
        fs id:          767465ce-3031-4672-8341-dfb135d8a463
        session 0:
                mount point:    broze:/snaps
                device:         broze:/dev/mapper/vg0-snaps
                time:           Tue Nov  3 14:23:57 2009
                session label:  "dump1"
                session id:     531a3622-84af-4767-a54b-a1b11a962dcb
                level:          0
                resumed:        NO
                subtree:        YES
                streams:        1
                stream 0:
                        pathname:       /dev/sde2
                        start:          ino 535 offset 0
                        end:            ino 1260 offset 0
                        interrupted:    YES
                        media files:    1
                        media file 0:
                                mfile index:    0
                                mfile type:     data
                                mfile size:     750000128
                                mfile start:    ino 535 offset 0
                                mfile end:      ino 1260 offset 0
                                media label:    "drive1"
                                media id:       721c35ba-e844-47f8-8692-0d3122d88093
xfsdump: Dump Status: SUCCESS

Seems to indicate I should be able to resume the backup. However, if I run xfsdump with the -R flag and specify a new device to backup to, this is what I get:

# xfsdump -R -o -l0 -s daily.0/jones -f /dev/sde1 -p 10 /snaps
xfsdump: using file dump (drive_simple) strategy
xfsdump: version 2.2.45 (dump format 3.0) - Running single-threaded

 ============================= dump label dialog ==============================

please enter label for this dump session (timeout in 300 sec)
 -> dump1.contd
session label entered: "dump1.contd"

 --------------------------------- end dialog ---------------------------------

xfsdump: ERROR: resume (-R) option inappropriate: no interrupted level 0 dump to resume
xfsdump: Dump Status: ERROR

The /dev/sde2 partition does appear to contain a valid, if incomplete, backup as I can restore from it using the xfsrestore command.

Any idea how I can get xfsdump to span multiple devices? Is there a better way to accomplish this without manually trying to partition the data into 2TB chunks?

Thanks!

I have a RAID 1 + hot spare that I would like to reconstruct, online, into a RAID 5 set. I know this is possible with OpenManage, but I has anyone figured out how to pull this off using the MegaCli tools?

I manage around 30 Ubuntu servers using puppet. I've seen many references to cron-apt and apticron as approaches to keeping their packages up to date but I haven't been able to find a way to centrally manage the process. With cront-apt/apticron I would still need to log in to each host and run aptitude update to perform the update. Not to mention review notifications from all 30 machines whenever a core package is updated.

There has to be a better way. Any suggestions?

I'm using a MultiTech GPRS modem on Ubuntu 8.0.4 (2.6.24-24 kernel with the MultiTech patches to the ti_usb_3410_5052 module).

The device works great. However if I disconnect it and then reconnect without removing the module in between, it hangs. Attempting to modprobe -r ti_usb_3410_5052 at this point hangs. And running lsusb hangs.

So. There must be a way to trigger the removal of the module when I disconnect the device. Any hints? Maybe a udev rule?

PCI-DSS 3.0 requirement 8.1.8 states: "If a session has been idle for more than 15 minutes, require the user to re-authenticate to re-activate the terminal or session." Same was in PCI-DSS 2.0 requirement 8.5.15.

The first, and most obvious, way to deal with ssh sessions that are idling at the bash prompt is by enforcing a read-only, global $TMOUT of 900. Unfortunately, that only covers sessions sitting at the bash prompt. The spirit of the PCI spec would also require killing sessions running top/vim/etc.

I've considered writing a */1 cron job that parses the output of "/usr/bin/w" and kills the associated shell, but that seems like a blunt instrument. Any ideas for something that would actually do what the spec requires and just lock the terminal? I've looked at away and vlock; they both seem great for voluntarily locking your terminal, but I need a cron/daemon task that will enforce locking.