Home / server / Questions / 628956
Accepted
Insyte
Asked: 2014-09-17 08:43:39 +0800 CST

Properly ordering shutdown of MySQL and iSCSI on Ubuntu Lucid

  • 772

I'm running MySQL on an Ubuntu 10.04 server, with the MySQL $datadir located on an iSCSI volume. The filesystem is tagged with "_netdev" in /etc/fstab. On system shutdown, upstart sends MySQL a SIGTERM, then moves on to shutting down the network interfaces, without waiting for MySQL to exit or for the iSCSI disk to be unmounted or disconnected. This, of course, results in XFS not unmounting cleanly and MySQL needing to spend several minutes replaying InnoDB logs on startup.

I'm not clear on the right place to begin poking to fix this. From what I've read, upstart should already be smart enough to wait for iSCSI filesystems to unmount and disconnect before it shuts down the network interfaces, although this isn't happening. And I have no idea how to tell it that it also needs to wait for a clean shutdown of MySQL.

This is what the console shows during the shutdown process:

Broadcast message from root@mysql-a1
    (unknown) at 11:19 ...

The system is going down for halt NOW!
Power button pressed
 * Stopping Bacula File daemon...                                        [ OK ]
Stopping file integrity checker: samhain.
 * Running nssldap-update-ignoreusers...                                 [ OK ]
 * Stopping multipath daemon multipathd                                  [ OK ]
 * Stopping nagios-nrpe nagios-nrpe                                      [ OK ]
 * Stopping Name Service Cache Daemon nscd                               [ OK ]
 * Stopping Postfix Mail Transport Agent postfix                         [ OK ]
 * Stopping SSH throttling throttle-ssh                                  [ OK ]
 * Stopping puppet agent                                                 [ OK ]
 * Stopping system logging syslog-ng                                     [ OK ]
Stopping statistics collection and monitoring daemon: collectdcollectd[1210]: Exiting normally.
collectd[1210]: collectd: Stopping 15 read threads.
collectdmon[1209]: Info: collectd terminated with exit status 0
collectdmon[1209]: Info: shutting down collectdmon
.
 * Asking all remaining processes to terminate...                        [ OK ]
 * All processes ended within 1 seconds....                              [ OK ]
 * Deconfiguring network interfaces...                                          [ 2884.248199] end_request: I/O error, dev dm-0, sector 5216
[ 2884.249807] end_request: I/O error, dev dm-0, sector 4192
[ 2884.817855] end_request: I/O error, dev dm-0, sector 0
[ 2884.819347] XFS (dm-0): Device dm-0: metadata write error block 0x0
[ 2884.821281] XFS (dm-0): I/O Error Detected. Shutting down filesystem
[ 2884.823393] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[ 2885.367423] end_request: I/O error, dev dm-0, sector 0
init: mysql main process (4168) terminated with status 1
                                                                         [ OK ]
 * Deactivating swap...                                                  [ OK ]
 * Unmounting weak filesystems...                                        [ OK ]
 * Unmounting local filesystems...                                       [ OK ]
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Will now halt
[ 2886.802324] Power down.
Connection to bottom.cw closed.
bottom:~ insyte$

This is the fstab entry:

/dev/mapper/21db3d79bf30ef4846c9ce90069680087 /srv/mysql xfs  _netdev,noatime,nodev,noexec  0 0
ubuntu

1 Answers

  1. Best Answer

    You have a couple options:

    1. Upgrade to precise. The issue is fixed there.
    2. Change the stop on condition (in /etc/init/mysql.conf) to stop on starting rc RUNLEVEL=[016]
    3. Go into whatever init script (I am pretty sure it is sysv) brings down the network interfaces, and add the command initctl emit deconfiguring-networking just before the interfaces are brought down. Then, change the stop on condition of mysql to stop on deconfiguring-networking or runlevel [016]

    The middle option is probably simplest.

    • 0