shlomoid's questions

When running DTT iotop on a write heavy Solaris 10 server, which runs multiple zones with MySQL daemons installed, I get the following output:

  UID    PID   PPID CMD              DEVICE  MAJ MIN D            BYTES
   70  26636      1 mysqld           sd1      10  64 R           360448
   70  25940      1 mysqld           sd1      10  64 R           530432
    0      5      0 zpool-rpool      sd1      10  64 W         17250816

What bothers me here is the fact that zpool-rpool takes up most of the io. What can I do to see which of the MySQL or other processes really takes up the IO - a more elaborate breakdown? If zpool-rpool represents "writes to ZFS", then iotop is really not helping me here... :)

Thanks!

The title is a bit confusing, but I can't think of a better one.

What I have is a simple vanilla MySQL replication, with the slave occasionally failing, with this error: Error 'Can't find record in 'my_tbl'' on query. Default database: 'my_db'. Query: 'UPDATE my_tbl SET ... WHERE ...' (columns omitted for clarity).

What I'm assuming this error means, is that the slave sql thread executed this update, and received 0 rows affected. This was not what it expected when comparing the result of 1 rows affected from the relay log, thus generating an error.

When running this same update transaction manually, it works. Same thing when running START SLAVE - it just starts working, and goes back to normal.

This doesn't make sense to me at all - if all it takes is a "retry" to fix this, how could this happen in the first place? Everything is executed in a serialized fashion, and nothing else is writing to the slave mysql server.

Can someone provide an explanation?

Some technicalities - this is a mixed replication setup from 5.5.7-rc to 5.5.12.

When running "echo ::memstat | mdb -k" I get this result:

Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    2349789              9178   14%
ZFS File Data             4251652             16608   25%
Anon                      8101424             31646   48%
Exec and libs               51583               201    0%
Page cache                1018684              3979    6%
Free (cachelist)             5049                19    0%
Free (freelist)            992659              3877    6%

Total                    16770840             65511
Physical                 16323297             63762

Everything I read online says that the the file system caching layer (ARC) in ZFS is "ZFS File Data". What does "Page cache" stand for? Why is it so large then?

I'm sorry if this is a basic question, but no google search I tried turned up any reasonable results.

I have a default Solaris 10 installation running on a 64GB box. For some reason, not all of the memory is available for use. The reason I'm even bothered by it, is because I'm trying to get mysql to run there, and give it 58GB of RAM, or more, but the maximum I can map is 45GB (nmap fails).

Nothing is running, except the GUI (which we will remove soon). But the UI doesn't explain the fact that 13 GB of the memory are gone, somewhere.

If ARC is a part of the used memory in Solaris (as opposed to the filecache in linux), then the UI consumes 5GB, and we still should have at least 55GB of RAM for MySQL. BUT: if this memory is consumed by the ARC, it should give the memory up, when innodb tries to nmap the memory (or does it?), so it makes no sense.

How can I see what consumes this memory, and free it?

Some technical data:

#kstat -m zfs | grep size
    data_size                       7847447040
    hdr_size                        139495272
    l2_hdr_size                     0
    l2_size                         0
    other_size                      510852000
    size                            8497794312

# top
96 processes: 95 sleeping, 1 on cpu
CPU states: 99.9% idle,  0.0% user,  0.1% kernel,  0.0% iowait,  0.0% swap
Memory: 64G phys mem, 51G free mem, 2048M swap, 2048M free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  1529 root       1  59    0   67M   10M sleep   11:17  0.20% mixer_applet2
   902 root       1  59    0   12M 9848K sleep    4:33  0.08% gconfd-2
  1923 root      34  59    0  165M   57M sleep    3:50  0.06% java
  1525 root       1  59    0   69M   13M sleep    3:24  0.06% gnome-netstatus
   644 root       1  58    0   45M   29M sleep    2:38  0.05% Xorg
  1501 root       1  59    0   73M   18M sleep    2:16  0.04% gnome-panel
 19814 root       1  59    0 3268K 1820K cpu      0:00  0.03% top
  2416 root      47  59    0   87M   68M sleep    2:08  0.03% java
 22587 noaccess  18  59    0  131M  115M sleep    2:13  0.03% java
   744 noaccess  18  59    0  132M  115M sleep    2:13  0.03% java
   908 root       1  59    0 6608K 4020K sleep    0:09  0.01% xscreensaver
  2435 root       2  49    0   73M   14M sleep    0:18  0.01% gnome-terminal
  2497 root      23  59    0   11M 7384K sleep    0:04  0.01% nscd
 22367 root       1  59    0 9832K 6636K sleep    0:01  0.00% snmpd
 15253 root       1  59    0 6516K 3840K sleep    0:00  0.00% sshd

# prtdiag -v
System Configuration: Dell Inc. PowerEdge R710
BIOS Configuration: Dell Inc. 2.2.10 11/09/2010
BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)

==== Processor Sockets ====================================

Version                          Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz CPU1
Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz CPU2

==== Memory Device Sockets ================================

Type    Status Set Device Locator      Bank Locator
------- ------ --- ------------------- --------------------
Unknown in use 1   DIMM_A1             
Unknown in use 1   DIMM_A2             
Unknown empty  2   DIMM_A3             
Unknown in use 2   DIMM_A4             
Unknown in use 3   DIMM_A5             
Unknown empty  3   DIMM_A6             
Unknown empty  4   DIMM_A7             
Unknown empty  4   DIMM_A8             
Unknown empty  5   DIMM_A9             
Unknown in use 5   DIMM_B1             
Unknown in use 6   DIMM_B2             
Unknown empty  6   DIMM_B3             
Unknown in use 4   DIMM_B4             
Unknown in use 5   DIMM_B5             
Unknown empty  6   DIMM_B6             
Unknown empty  4   DIMM_B7             
Unknown empty  5   DIMM_B8             
Unknown empty  6   DIMM_B9             

==== On-Board Devices =====================================
Embedded Matrox G200 Video
Embedded Broadcom 5709C NIC 1
Embedded Broadcom 5709C NIC 2
Embedded Broadcom 5709C NIC 3
Embedded Broadcom 5709C NIC 4
Integrated RAID Controller

==== Upgradeable Slots ====================================

ID  Status    Type             Description
--- --------- ---------------- ----------------------------
1   available Unknown          PCI1
2   available Unknown          PCI2
3   in use    Unknown          PCI3
4   available Unknown          PCI4

#prstat -a
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
 21369 root     3888K 3036K cpu10   59    0   0:00:00 0.0% prstat/1
  1529 root       67M   10M sleep   59    0   0:11:22 0.0% mixer_applet2/1
 21234 100        45G 3686M sleep   59    0   0:00:17 0.0% mysqld/25
   902 root       12M 9848K sleep   59    0   0:04:35 0.0% gconfd-2/1
   644 root       45M   29M sleep   59    0   0:02:39 0.0% Xorg/1
   642 root     2560K  764K sleep   59    0   0:00:00 0.0% fbconsole/1
   640 root     5704K 1408K sleep   59    0   0:00:00 0.0% dtlogin/1
   863 root     4872K 1968K sleep   59    0   0:00:00 0.0% sdt_shell/1
  2438 root     1432K  832K sleep   59    0   0:00:00 0.0% sh/1
  2436 root     2644K 1392K sleep   59    0   0:00:00 0.0% gnome-pty-helpe/1
   883 root     1532K  892K sleep   59    0   0:00:00 0.0% Xsession2.jds/1
  2435 root       73M   14M sleep   49    0   0:00:18 0.0% gnome-terminal/2
   789 root     1568K  940K sleep   59    0   0:00:00 0.0% Xsession/1
   783 smmsp    9452K 2120K sleep   59    0   0:00:00 0.0% sendmail/1
   864 root     3228K  836K sleep   59    0   0:00:00 0.0% dsdm/1
   435 root     2064K  940K sleep   59    0   0:00:00 0.0% smcboot/1
   436 root     2064K  684K sleep   59    0   0:00:00 0.0% smcboot/1
   402 root     1432K  624K sleep   59    0   0:00:00 0.0% utmpd/1
   866 root     1456K  840K sleep   59    0   0:00:00 0.0% sh/1
   437 root     2064K  684K sleep   59    0   0:00:00 0.0% smcboot/1
   407 root     3988K 1936K sleep   59    0   0:00:00 0.0% syslogd/11
   630 root     3256K 2260K sleep   59    0   0:00:00 0.0% vold/5
  2497 root       11M 7392K sleep   59    0   0:00:04 0.0% nscd/23
 22200 root     3876K 1568K sleep   59    0   0:00:00 0.0% syslogd/13
   399 root     7996K 3996K sleep   59    0   0:00:01 0.0% inetd/4
   415 root     2468K 1372K sleep   59    0   0:00:00 0.0% ttymon/1
   404 root     2504K 1276K sleep   59    0   0:00:00 0.0% ttymon/1
   295 daemon   2932K 1432K sleep   59    0   0:00:00 0.0% rpcbind/1
   142 root     5452K 2184K sleep   59    0   0:00:00 0.0% syseventd/15
   224 root     2824K 1080K sleep   59    0   0:00:00 0.0% cron/1
   156 root     4344K 3164K sleep   59    0   0:00:00 0.0% picld/9
   387 root       18M   14M sleep   59    0   0:00:02 0.0% fmd/19
   152 daemon   4612K 2684K sleep   59    0   0:00:00 0.0% kcfd/4
   397 root     2132K 1040K sleep   59    0   0:00:00 0.0% sac/1
 19861 root     5864K 3720K sleep   59    0   0:00:00 0.0% zlogin/1
   204 root     2244K  780K sleep   59    0   0:00:00 0.0% iscsi-initiator/2
  2498 daemon   2716K 1676K sleep   59    0   0:00:00 0.0% statd/1
   369 root     2528K 1292K sleep   59    0   0:00:03 0.0% in.routed/1
    77 root     6148K 3484K sleep   59    0   0:00:02 0.0% devfsadm/9
  1923 root      165M   57M sleep   59    0   0:03:51 0.0% java/34
   669 root     6600K 2912K sleep   59    0   0:00:00 0.0% dtlogin/1
    11 root       12M   11M sleep   59    0   0:00:08 0.0% svc.configd/16
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    91 root      423M  425M   0.6%   0:34:57 0.0%
     1 100        45G 3703M   5.7%   0:00:17 0.0%
     2 noaccess  251M  237M   0.4%   0:04:28 0.0%
     2 smmsp    3804K   16M   0.0%   0:00:00 0.0%
    10 daemon     10M   14M   0.0%   0:00:00 0.0%

#  echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     646470              2525    4%
ZFS File Data             1920208              7500   11%
Anon                      1078294              4212    6%
Exec and libs               14431                56    0%
Page cache                   2544                 9    0%
Free (cachelist)            13012                50    0%
Free (freelist)          13096905             51159   78%

Total                    16771864             65515
Physical                 16324293             63766

We have a busy server, chocking under a high IO load, at least, that's the feeling I have. Output from iostat -xz looks like this:

             extended device statistics                 
device    r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b 
sd5     224.8  157.8 10701.8 6114.7  0.0  9.5   24.7   0 100 
sd5     243.2  110.4 11565.3 4065.0  0.0  9.7   27.5   0 100

It's obvious that the disk subsystem is overloaded, since a 25ms service time is unacceptable for a 6 drive SATA array, and a 100% busy also means we're chocked on disk IO.

But - why is wait always 0.0? And why is %w is also 0? %w sometimes goes to 1, and quickly returns to 0. Doesn't this mean that no process is waiting for IO?

Does the RAID controller somehow causes this result / masks the wait times?

Can someone explain this behavior?

I'm trying to enable automatic ZFS snapshotting on a Solaris 10 server. From extensive googling, I've managed to understand that there are/were some automated services to do this, like:

• http://mij.oltrelinux.com/devel/zfsbackup/

And even locate a manual of sorts for a built in service:

• http://download.oracle.com/docs/cd/E19120-01/open.solaris/817-2271/6mhupg6oc/index.html

Alas, on the Solaris I'm working on, I can't seem to find the "Time Slider" service, nor the auto-snapshot service other articles refer to, that is supposed to be built in, and simply enabled.

Does anyone know how I can install/enable this service?