Pavel's questions

I have an application that mmaps a large read-only data structure into memory and reads parts of it. After some time, around 1 GB of physical memory is filled with pages loaded from this data structure. When the amount of free memory on that machine drops below a certain threshold (~750 MB or so), the kernel seems to release all (or almost all) pages loaded through mmap.

How does Kernel decide when to start unloading mmapped pages? Is there a way to control that threshold? (e.g. anything in /proc/sys/vm or cgroups)

I have a global cgroup defined in /etc/cgconfig.conf that limits the amount of memory. Everytime a user runs a command, I prepend cgexec to add the process and its children to the controlled group. Every now and then the limit kicks in and kills the user process.

If the exit code is not 0, how do I know if the process just failed because of some internal logic, or if it has been killed by the cgroup mechanism?

It's running in user space, so I'd like to avoid parsing /var/log/syslog.

So I'm connecting to a VPN server from Ubuntu 16.04 via

$ sudo openvpn client.ovpn

And client.ovpn starts with

# Automatically generated OpenVPN client config file
# Generated on Mon Jun  6 10:36:29 2016 by openvpnas
# Note: this configuration is user-locked to the username below
...
# OVPN_ACCESS_SERVER_CLI_PREF_ALLOW_WEB_IMPORT=True
# OVPN_ACCESS_SERVER_CLI_PREF_BASIC_CLIENT=False
# OVPN_ACCESS_SERVER_CLI_PREF_ENABLE_CONNECT=True
# OVPN_ACCESS_SERVER_CLI_PREF_ENABLE_XD_PROXY=True
...
# OVPN_ACCESS_SERVER_WEB_CA_BUNDLE_START
# -----BEGIN CERTIFICATE-----
...
# -----END CERTIFICATE-----
# OVPN_ACCESS_SERVER_WEB_CA_BUNDLE_STOP
# OVPN_ACCESS_SERVER_IS_OPENVPN_WEB_CA=1
# OVPN_ACCESS_SERVER_ORGANIZATION=OpenVPN Technologies, Inc.
setenv FORWARD_COMPATIBLE 1
client
server-poll-timeout 4
nobind
...
dev tun
dev-type tun
ns-cert-type server
reneg-sec 604800
sndbuf 100000
rcvbuf 100000
auth-user-pass
# NOTE: LZO commands are pushed by the Access Server at connect time.
# NOTE: The below line doesn't disable LZO.
comp-lzo no
verb 3
setenv PUSH_PEER_INFO

The connection terminates after 24 hours automatically. How do I make it persistent? Is there an option for that I can pass to openvpn?

Here's one of my Solaris 10 servers after a couple of disk replacements in a zpool

  pool: volume
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Wed Jul  6 11:43:51 2016
    207M scanned out of 36.7T at 1.48M/s, (scan is slow, no estimated time)
    207M scanned out of 36.7T at 1.48M/s, 7235h37m to go
    13.5M resilvered, 0.00% done
config:

        NAME         STATE     READ WRITE CKSUM
        volume       ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            c4t0d0   ONLINE       0     0     1
            c4t0d1   ONLINE       0     0     0
            c4t0d2   ONLINE       0     0     0
            c4t0d3   ONLINE       0     0     0
            c4t0d22  ONLINE       0     0     0
            c4t0d5   ONLINE       0     0     0
            c4t0d6   ONLINE       0     0     0
            c4t0d23  ONLINE       0     0     0  (resilvering)
            c4t0d15  ONLINE       0     0     0
            c4t0d9   ONLINE       0     0     0
            c4t0d10  ONLINE       0     0     0
          raidz2-1   ONLINE       0     0     4
            c4t0d11  ONLINE       0     0     0
            c4t0d8   ONLINE       0     0     0  (resilvering)
            c4t0d13  ONLINE       0     0     0
            c4t0d14  ONLINE       0     0     0
            c4t0d20  ONLINE       0     0     0
            c4t0d16  ONLINE       0     0     0
            c4t0d4   ONLINE       0     0     0
            c4t0d18  ONLINE       0     0     2
            c4t0d19  ONLINE       0     0     0
            c4t0d17  ONLINE       0     0     0
            c4t0d21  ONLINE       0     0     0

errors: No known data errors

The scan status returns to 0.00% done every 10-15 minutes, thus re-starting the resilvering. Here is the output of echo "::zfs_dbgmsg" | mdb -k.

iostat -En shows a high (and growing) number of errors on all disks.

zpool iostat -v volume shows normal resilvering activity (write on new disks, read from old disks)

/var/adm/messages is full of messages like this:

Jul  6 12:08:25 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1/sd@0,15 (sd20):
Jul  6 12:08:25 raid2   SCSI transport failed: reason 'reset': retrying command
Jul  6 12:08:28 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1/sd@0,b (sd8):
Jul  6 12:08:28 raid2   Error for Command: read(10)                Error Level: Retryable
Jul  6 12:08:28 raid2 scsi: [ID 107833 kern.notice]     Requested Block: 21523458                  Error Block: 21523458
Jul  6 12:08:28 raid2 scsi: [ID 107833 kern.notice]     Vendor: transtec                           Serial Number: 63881076-00 
Jul  6 12:08:28 raid2 scsi: [ID 107833 kern.notice]     Sense Key: Unit Attention
Jul  6 12:08:28 raid2 scsi: [ID 107833 kern.notice]     ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0
Jul  6 12:09:35 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1 (mpt0):
Jul  6 12:09:35 raid2   Disconnected command timeout for Target 0
Jul  6 12:09:39 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1/sd@0,15 (sd20):
Jul  6 12:09:39 raid2   incomplete read- retrying
Jul  6 12:10:46 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1 (mpt0):
Jul  6 12:10:46 raid2   Disconnected command timeout for Target 0
Jul  6 12:10:49 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1/sd@0,e (sd11):
Jul  6 12:10:49 raid2   incomplete read- retrying
Jul  6 12:11:56 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1 (mpt0):
Jul  6 12:11:56 raid2   Disconnected command timeout for Target 0
Jul  6 12:13:03 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1/sd@0,1 (sd35):
Jul  6 12:13:03 raid2   Error for Command: write                   Error Level: Retryable
Jul  6 12:13:03 raid2 scsi: [ID 107833 kern.notice]     Requested Block: 644                       Error Block: 644
Jul  6 12:13:03 raid2 scsi: [ID 107833 kern.notice]     Vendor: transtec                           Serial Number: 023CEC5B-00 
Jul  6 12:13:03 raid2 scsi: [ID 107833 kern.notice]     Sense Key: Unit Attention
Jul  6 12:13:03 raid2 scsi: [ID 107833 kern.notice]     ASC: 0x29 (power on, reset, or bus reset occurred), ASCQ: 0x0, FRU: 0x0
Jul  6 12:13:03 raid2 scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,3597@4/pci8086,329@0/pci1000,1060@1/sd@0,5 (sd2):

Is there anything I can do to make resilvering finish so that the pool can return to a normal state?

Currently we cannot fill the 19" rack completely. So we have 12 GPU nodes, 2U each. The 42U rack is connected to a cooling unit that blows cold air in front of the computers, sucking in hot air from behind. As far as I understood this thread, the best practice is to stack all nodes without leaving any gaps in between and to use blank panels to close all empty slots.

Now where in the rack should I put this 24U stack of hot computers? All at the bottom, to let the cool air sink to the lowest node? All to the top, because the wind flow is strong enough? Somewhere in the middle?

I know I can use qacct to get all kind of statistics about running jobs. Now is there a way get some statistics about how many jobs are pending and how long do they need to wait on average?

Bonus question: can I get statistics about "idling" (or empty) slots in SGE?

I'm running ZFS on a Ubuntu machine that hosts our home directories. The pool has the following ACL setting:

# zfs get all homes | grep acl
homes  aclinherit            restricted             default
homes  acltype               off                    default

Here I have two regular files with the same permissions:

# getfacl 1.txt 2.txt 
# file: 1.txt
# owner: usr
# group: grp
user::rw-
group::r--
other::r--

# file: 2.txt
# owner: usr
# group: grp
user::rw-
group::r--
other::r--

Every night I send an incremental snapshot to a machine running ZFS on Solaris 8

zfs send -i homes@$PREV_BACKUP homes@$CURRENT_BACKUP | \
   ssh solarishost zfs receive -vFd homes_backup

The file system on the Solaris host has the following ACL settings:

# zfs get all homes_backup | grep acl
homes_backup  aclmode               passthrough            local
homes_backup  aclinherit            restricted             default

When I inspect the files in the snapshot on the Solaris host, I see that the more recent one (1.txt) does not have any ACLs set:

# /usr/bin/ls -v 1.txt 2.txt 
-rw-r--r--   1 2428     2000        2170 Oct 12 13:42 1.txt
-rw-r--r--   1 2428     2000        2146 May 31  2013 2.txt
     0:owner@:execute:deny
     1:owner@:read_data/write_data/append_data/write_xattr/write_attributes
         /write_acl/write_owner:allow
     2:group@:write_data/append_data/execute:deny
     3:group@:read_data:allow
     4:everyone@:write_data/append_data/write_xattr/execute/write_attributes
         /write_acl/write_owner:deny
     5:everyone@:read_data/read_xattr/read_attributes/read_acl/synchronize
         :allow

Now when I mount the solarishost-snapshot on a Ubuntu client, I can see the files and they appear to have identical permissions, but I cannot read 1.txt:

# cat 2.txt >/dev/null; echo $?
0
# cat 1.txt >/dev/null; echo $?
cat: 1.txt: Permission denied
1

Again, the look the same to the Ubuntu client:

# getfacl 1.txt 2.txt
# file: 1.txt
# owner: usr
# group: grp
user::rw-
group::r--
mask::rwx
other::r--

# file: 2.txt
# owner: usr
# group: grp
user::rw-
group::r--
mask::rwx
other::r--

# ls -l 1.txt 2.txt 
-rw-r--r--+ 1 usr grp 2.2K Oct 12 13:42 1.txt
-rw-r--r--+ 1 usr grp 2.1K May 31  2013 2.txt

My question is now: how do I set up the snapshot creation from Ubuntu to Solaris such that the ACLs on the Solaris host allow users to mount the snapshots and read their own files? Looks like all recently backed up files are affected.

TL;DR: I don't understand why I can access one file but not an other (in the same directory), although they appear to have identical permissions.

I have an NFS server running Solaris 10 10/08 and a client on Ubuntu 12.04. I mount a ZFS snapshot like this:

# mount | grep homes
server:/homes/.zfs/snapshot on /u/homes type nfs (ro,noatime,vers=3,intr,sloppy,addr=1.2.3.4)

Then I look at these two files in /u/homes:

# ls -l 1.txt 2.txt 
-rw-r--r--+ 1 usr grp 2.2K Oct 12 13:42 1.txt
-rw-r--r--+ 1 usr grp 2.1K May 31  2013 2.txt


# stat 1.txt
  File: `1.txt'
  Size: 2170        Blocks: 4          IO Block: 1048576 regular file
Device: 43h/67d Inode: 1621536     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 2428/   usr)   Gid: ( 2000/    grp)
 Birth: -

# stat 2.txt
  File: `2.txt'
  Size: 2146        Blocks: 3          IO Block: 1048576 regular file
Device: 43h/67d Inode: 7975730     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 2428/   usr)   Gid: ( 2000/    grp)
 Birth: -

Same files when viewed on the server:

  File: `1.txt'
  Size: 2170            Blocks: 4          IO Block: 2560   regular file
Device: 2d50005h/47513605d      Inode: 1621536     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 2428/ UNKNOWN)   Gid: ( 2000/ UNKNOWN)

  File: `2.txt'
  Size: 2146            Blocks: 3          IO Block: 2560   regular file
Device: 2d50005h/47513605d      Inode: 7975730     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 2428/ UNKNOWN)   Gid: ( 2000/ UNKNOWN)

They look very similar to me. Now when I try to access them from the client, I get

# cat 2.txt >/dev/null; echo $?
0
# cat 1.txt >/dev/null; echo $?
cat: 1.txt: Permission denied
1

How's that? They have identical permissions! I also snoop-ed the NFS packets on the server side and there was no difference in communication (other than packet lengths, checksums, timestamps etc.), but one file returns

NFS:  Access = read 

and the other

NFS:  Access =

How can this be and what am I doing wrong?

EDIT1

# zfs get all homes | sort
NAME   PROPERTY              VALUE                       SOURCE
homes  aclinherit            restricted                  default
homes  aclmode               groupmask                   default
homes  atime                 on                          default
homes  available             1.36T                       -
homes  canmount              on                          default
homes  casesensitivity       sensitive                   -
homes  checksum              on                          default
homes  compression           on                          inherited from homes
homes  compressratio         1.57x                       -
homes  copies                1                           default
homes  creation              Mon Aug 17 18:53 2015       -
homes  devices               on                          default
homes  exec                  on                          default
homes  logbias               latency                     default
homes  mounted               yes                         -
homes  mountpoint            /homes                      local
homes  nbmand                off                         default
homes  normalization         none                        -
homes  primarycache          all                         default
homes  quota                 none                        default
homes  readonly              off                         inherited from homes
homes  recordsize            128K                        default
homes  referenced            800G                        -
homes  refquota              none                        default
homes  refreservation        none                        default
homes  reservation           none                        default
homes  secondarycache        all                         default
homes  setuid                on                          default
homes  shareiscsi            off                         default
homes  sharenfs              on                          inherited from homes
homes  sharesmb              off                         default
homes  snapdir               hidden                      default
homes  type                  filesystem                  -
homes  used                  1.31T                       -
homes  usedbychildren        0                           -
homes  usedbydataset         800G                        -
homes  usedbyrefreservation  0                           -
homes  usedbysnapshots       544G                        -
homes  utf8only              off                         -
homes  version               4                           -
homes  vscan                 off                         default
homes  xattr                 on                          default
homes  zoned                 off                         default


# zfs get all homes@backup_2016-01-21 | sort
NAME                      PROPERTY              VALUE                  SOURCE
homes@backup_2016-01-21  casesensitivity       sensitive              -
homes@backup_2016-01-21  compressratio         1.53x                  -
homes@backup_2016-01-21  creation              Thu Jan 21  0:00 2016  -
homes@backup_2016-01-21  defer_destroy         off                    -
homes@backup_2016-01-21  devices               on                     default
homes@backup_2016-01-21  exec                  on                     default
homes@backup_2016-01-21  nbmand                off                    default
homes@backup_2016-01-21  normalization         none                   -
homes@backup_2016-01-21  primarycache          all                    default
homes@backup_2016-01-21  referenced            800G                   -
homes@backup_2016-01-21  secondarycache        all                    default
homes@backup_2016-01-21  setuid                on                     default
homes@backup_2016-01-21  shareiscsi            off                    default
homes@backup_2016-01-21  type                  snapshot               -
homes@backup_2016-01-21  used                  0                      -
homes@backup_2016-01-21  userrefs              0                      -
homes@backup_2016-01-21  utf8only              off                    -
homes@backup_2016-01-21  version               4                      -
homes@backup_2016-01-21  xattr                 on                     default

EDIT2

# share
-               /homes   rw   ""  

I can't find the driver for PERC H730P Mini for Solaris 10. The installation in a Dell PowerEdge R730xd fails with "no disks found".

Isn't this controller supported? Where can I get a compatible driver?

Thanks!

So we have a bunch of SSD disks and would like to put them in a zpool on a Solaris 10 system. The file system will be exported via NFS to about 50 Ubuntu clients which will mount it as their $HOME. I expect that the bandwidth will not exceed 1 Gbit/s, but the latency should be as low as possible because of the Desktop environments running on the clients.

1. What is a good configuration for such a zpool? Currently we have one raidz2 with 8 disks + 2 hot spares, but I've read that since a single raidz is just a single vdev, the performance is limited to the speed of a single disk.
2. What are the critical NFS server/client parameters that can be tuned? Currently we use NFS3 with noatime and default rsize/wsize, but perhaps there is a better choice for the clients running on Ubuntu.

We're considering to equip out server farm with water cooled rack doors, similar to

• http://www.emersonnetworkpower.com/en-US/Products/PrecisionCooling/HighDensityModularCooling/Water-Based/Pages/LiebertDCD.aspx
• http://www.motivaircorp.com/products/high-density-server-rack-cooling/chilled-door
• https://www.electrorack.com/passive-water-cooled-door/HydroCool-IT.htm

since this is a pretty specific type of hardware, it's hard to find any information on those things that is independent from vendors.

In the most simple setting, we would buy a single read-door-cooled rack connected to a Coolant Distribution Unit (CDU) that provides about ~20 kW of cooling capacity. In this rack we would like to deploy

• option 1: 3 Blade enclosures (x16 nodes) or
• option 2: 2 enclosures + several GPU nodes

Both options will produce about 17 kW of heat. All dimensionality questions aside (water/room temperature, water flow, pressure, etc), does anyone of you have experience with such cooling rack doors and could share how well they works in a room that has approx. 20°C air temperature? The vendors promise t_out = t_in, but I'm curious whether it holds true in case of a really hot hardware. My concerns include:

• the heat is not uniformly distributed over the door area, since the blades have those hot spots where the fans are blowing lots of hot air
• how strong does a cooling door slow down the air flow? isn't there a chance of air congestion?
• is it possible for the CDU to adapt to changes in the room temperature? on a hot summer day, the room temperature may increase by a few degrees, so that it would be nice to have t_out < t_in.

The alternative is to set up a proper water-cooled rack that is completely closed and that pumps chilled air in front of the computers. however, those are typically somewhat more expensive. that's why I'm interested in your opinions on the read-door-cooling in the first place. Especially in case of very hot computers.

Thanks!

We have a large file server running on Solaris 10 and about 200 computing nodes running on Ubuntu. The nodes access the file server over NFS. The load gets larger and larger over time.

Is there a way to restrict the resources (either in/outbound network bandwidth or the I/O to/from the ZFS filesystem on the storage) on the per-user basis? If it's not possible on Solaris, can we tweak the network switch (Cisco Nexus 5k series) to achieve the same?

How do other people manage the access to a large single file system in a fair way?

I'm wondering why Solaris is reporting

action: Wait for the resilver to complete.
 scan: resilver in progress since Sun Jul 14 02:42:44 2013
    24.9T scanned out of 39.2T at 54.9M/s, 75h46m to go
    24.9T scanned out of 39.2T at 54.9M/s, 75h46m to go
    908G resilvered, 63.56% done

39.2T is the size of the complete pool, which is built from 4 raidz1 vdevs:

config:

        NAME                                       STATE     READ WRITE CKSUM
        volume                                     ONLINE       0     0     0
          raidz1-0                                 ONLINE       0     0     0
...
          raidz1-1                                 ONLINE       0     0     0
...
          raidz1-2                                 ONLINE       0     0     0
...
          raidz1-3                                 ONLINE       0     0     0
...

and every raidz1 has 7 disks x 2TB.

So, IIUC, the resilvering would have to go over <= 6x2TB to reconstruct the data of a single failed disk. How comes ZFS is scanning over the whole pool? I thought the redundant data needed for resilvering is not stored across multiple raidzs?

Thank you for explanation!

I know that the performance of ZFS heavily depends on the amount of free space:

Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. [...] Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer. ZFS_Best_Practices_Guide, solarisinternals.com (archive.org)

Now, suppose I have a raidz2 pool of 10T hosting a ZFS file system volume. Now I create a child file system volume/test and give it a reservation of 5T.

Then I mount both file systems per NFS to some host and perform some work. I understand that I can't write to volume more than 5T, because the remaining 5T are reserved to volume/test.

My first question is, how will the performance drop, if I fill my volume mount point with ~5T? Will it drop, because there is no free space in that file system for ZFS' copy-on-write and other meta-stuff? Or will it remain the same, since ZFS can use the free space within the space reserved for volume/test?

Now the second question. Does it make a difference, if I change the setup as follows? volume now has two file systems, volume/test1 and volume/test2. Both are given a 3T reservation each (but no quotas). Assume now, I write 7T to test1. Will the performance for both file systems be the same, or will it be different for every file system? Will it drop, or remain the same?

Thanks!

I'm about to configure some new GPU capable nodes for our needs and I was wondering if someone has any experience with using simple video cards like Nvidia GTX 680 (actively cooled with a fan) in a 1U server? The fan would be pretty close to the chassis (e.g. SuperMicro SC818G-1400B) and I'm not sure it will get enough cool air. In a usual ATX case the video card gets up to 80°C which is well below upper limit of 98°C.

Does anyone tried any similar configurations with activly cooled video cards in a 1U server and would recommend it?

Thanks!

I've configured a second host to replicate the main LDAP server via syncrepl in the slapd.conf:

syncrepl rid=666 
        provider=ldaps://my-main-server.com 
        type=refreshAndPersist 
        searchBase="dc=Staff,dc=my-main-server,dc=com" 
        filter="(objectClass=*)" 
        scope=sub 
        schemachecking=off 
        bindmethod=simple 
        binddn="cn=repadmin,dc=my-main-server,dc=com" 
        credentials=mypassword 

When I restart slapd, it writes to /var/log/debug

Jun 11 15:48:33 cluster-mn-04 slapd[29441]: @(#) $OpenLDAP: slapd 2.4.9 (Mar 31 2009 07:18:37) $ ^Ibuildd@yellow:/build/buildd/openldap2.3-2.4.9/debian/build/servers/slapd 
Jun 11 15:48:34 cluster-mn-04 slapd[29442]: slapd starting 
Jun 11 15:48:34 cluster-mn-04 slapd[29442]: null_callback : error code 0x14 
Jun 11 15:48:34 cluster-mn-04 slapd[29442]: syncrepl_entry: rid=666 be_modify failed (20) 
Jun 11 15:48:34 cluster-mn-04 slapd[29442]: do_syncrepl: rid=666 quitting 

I've looked into the sources for the return code and found only

#define LDAP_TYPE_OR_VALUE_EXISTS       0x14

in include/ldap.h. Anyway, I don't quite get what the error message means.

Can you help me debugging this problem and figure out why the LDAP replication doesn't work? I've managed to put a "manual" copy via slapcat and slapadd into the database, but I'd like to sync automatically.

UPDATE: "Solved" by removing /var/lib/ldap/* and re-importing the database with slapadd.

I'm still trying to get behind crashes on my Solaris 11 system. Recently, a core dump has been written during the crash. Looking into it with mdb yields

> $C
ffffff0021c09430 vpanic()
ffffff0021c09460 vcmn_err+0x2e(3, fffffffff7a8a830, ffffff0021c09520)
ffffff0021c09550 zfs_panic_recover+0xae()
ffffff0021c09610 dmu_buf_hold_array_by_dnode+0xbd(ffffff05207b5018, 400000, 20000, 0, fffffffff7a85ce0, 
ffffff0021c09654, ffffff0021c09658, 0)
ffffff0021c096b0 dmu_write_uio_dnode+0x50(ffffff05207b5018, ffffff0021c09a10, 20000, ffffff051efd0788)
ffffff0021c09700 dmu_write_uio_dbuf+0x58(ffffff05207b2320, ffffff0021c09a10, 20000, ffffff051efd0788)
ffffff0021c09960 zfs_write+0x843(ffffff051f120900, ffffff0021c09a10, 0, ffffff04e4b6adb0, 0)
ffffff0021c099d0 fop_write+0xa6(ffffff051f120900, ffffff0021c09a10, 0, ffffff04e4b6adb0, 0)
ffffff0021c09aa0 vn_rdwr+0x1bd(1, ffffff051f120900, ffffff051fb2e0c0, 20000, 400000, 1, 0, fffffffffffffffd, 
ffffff04e4b6adb0, ffffff0021c09ad8)
ffffff0021c09b20 zfs_replay_write+0xe3(ffffff04eb2b4200, ffffff051fb2e000, 0)
ffffff0021c09b60 zil_replay_wr_task+0x2d(ffffff04eb57ea00)
ffffff0021c09c00 taskq_thread+0x22e(ffffff051ede9810)
ffffff0021c09c10 thread_start+8()

How do I know what part of the stack caused the crash?

what can cause these kernel statistics (as reported by top) to be so high?

Kernel: 152661 ctxsw, 2475 trap, 99065 intr, 1449 syscall, 3 fork, 2373 flt

Usually, my system has much lower values, e.g.

Kernel: 487 ctxsw, 3 trap, 904 intr, 435 syscall, 3 flt

but every now and then the numbers go up and the OS freezes. The load is always <1.

Thank you!

Edit:

$ vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 s1 s3 s4   in   sy   cs us sy id
 1 0 0 2806128 2818224 43 207 0 0  0  0  5  4  4  0 10 9954  510 3740  0  2 98

$ prstat
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
   658 root        0K    0K sleep   60    -   2:15:27 0.5% nfsd_kproc/157
   245 root        0K    0K sleep   99  -20   0:16:08 0.2% zpool-volume/166
   577 root        0K    0K sleep   60    -   0:00:09 0.0% lockd_kproc/24
  8195 root       11M 4788K cpu0    49    0   0:00:00 0.0% prstat/1
   617 root       53M   36M cpu3    59    0   0:00:34 0.0% fmd/29
   117 root     2144K 1288K sleep   59    0   0:00:00 0.0% pfexecd/3
   136 root       13M 4824K sleep   59    0   0:00:00 0.0% syseventd/19
    46 root       17M 8260K sleep   59    0   0:00:01 0.0% dlmgmtd/20
    42 netcfg   3892K 2900K sleep   59    0   0:00:00 0.0% netcfgd/4
    94 daemon     14M 4824K sleep   59    0   0:00:00 0.0% kcfd/3
   614 daemon     12M 2068K sleep   59    0   0:00:00 0.0% nfsmapid/3
   708 hpsmh      24M 6256K sleep   59    0   0:00:00 0.0% hpsmhd/1
    13 root       19M   18M sleep   59    0   0:00:14 0.0% svc.configd/18
    11 root       24M   14M sleep   59    0   0:00:04 0.0% svc.startd/16
    71 netadm   4272K 2908K sleep   59    0   0:00:00 0.0% ipmgmtd/5
Total: 78 processes, 930 lwps, load averages: 0.40, 0.44, 0.46

Edit 2: stats just before the crash:

$ vmstat
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 s1 s3 s4   in   sy   cs us sy id
 1 0 0 2368992 2330108 41 216 0 0  0  0 130 4  4  0 64 39092 486 23076 0  7 93

$ prstat
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
   453 root        0K    0K sleep   99  -20   0:05:09 0.5% zpool-volume/166
   581 root        0K    0K sleep   60    -   0:20:36 0.4% nfsd_kproc/128
  1819 root       11M 6036K sleep   49    0   0:00:01 0.0% bash/1
   548 root        0K    0K sleep   60    -   0:00:16 0.0% lockd_kproc/12
     5 root        0K    0K sleep   99  -20   0:00:11 0.0% zpool-rpool/166
  1818 root       18M 5392K sleep   59    0   0:00:00 0.0% sshd/1
   555 root       58M   42M sleep   59    0   0:00:25 0.0% fmd/29
  3528 root       11M 5092K cpu5    59    0   0:00:00 0.0% prstat/1
     6 root        0K    0K sleep   99  -20   0:00:15 0.0% kmem_task/1
   501 root     9760K 1436K sleep   59    0   0:00:00 0.0% automountd/4
   499 root     9668K 1360K sleep   59    0   0:00:00 0.0% automountd/2
   488 root       14M 3896K sleep   59    0   0:00:00 0.0% inetd/4
   479 root     2780K 1488K sleep   59    0   0:00:00 0.0% hotplugd/3
   487 root     8928K 1164K sleep   59    0   0:00:00 0.0% cron/1
  1817 root       16M 3656K sleep   59    0   0:00:00 0.0% sshd/1
   468 daemon   7268K 4648K sleep   59    0   0:00:00 0.0% statd/1
   415 daemon   3508K 1440K sleep   59    0   0:00:02 0.0% rpcbind/1

I've got Solaris 11 (+latest SRU) running on an HP DL385 G7 (attached to P2000 storage w. 30 disks; they are registered as separate RAID0 drives, but I'm using ZFS' raidz1), which is our file server. Every couple of days, the system freezes and needs to be restarted. There is nothing special in the logs and fmdump.

I ended up with a cron job dumping various statistics every 2 minutes to the hard drive which show that there is a load increase and memory decrease just before the crash:

$ grep load top.120512*

top.120512063601:last pid: 21751;  load avg:  0.61,  2.30,  2.93;  up 4+17:03:45        06:36:02
top.120512063800:last pid: 21765;  load avg:  0.27,  1.62,  2.59;  up 4+17:05:44        06:38:01
top.120512064000:last pid: 21779;  load avg:  0.29,  1.17,  2.30;  up 4+17:07:45        06:40:02
top.120512064200:last pid: 21793;  load avg:  0.56,  0.97,  2.09;  up 4+17:09:44        06:42:01
top.120512064400:last pid: 21807;  load avg:  0.20,  0.71,  1.85;  up 4+17:11:45        06:44:02
top.120512064600:last pid: 21821;  load avg:  0.60,  0.66,  1.68;  up 4+17:13:45        06:46:02
top.120512064800:last pid: 21835;  load avg:  1.25,  0.87,  1.64;  up 4+17:15:44        06:48:01
top.120512065000:last pid: 21851;  load avg:  4.77,  2.35,  2.10;  up 4+17:17:45        06:50:02
top.120512065200:last pid: 21864;  load avg:  5.10,  3.20,  2.45;  up 4+17:19:45        06:52:02
top.120512065400:last pid: 21878;  load avg:  5.81,  4.16,  2.91;  up 4+17:21:44        06:54:01
top.120512065601:last pid: 21892;  load avg:  5.26,  4.53,  3.20;  up 4+17:23:45        06:56:02
top.120512065800:last pid: 21906;  load avg:  5.36,  4.79,  3.46;  up 4+17:25:45        06:58:02

// here was the crash

top.120512163801:last pid:   701;  load avg:  1.18,  0.29,  0.10;  up 0+00:01:16        16:38:02
top.120512164000:last pid:  1456;  load avg:  0.36,  0.33,  0.14;  up 0+00:03:16        16:40:02
top.120512164200:last pid:  1470;  load avg:  0.14,  0.26,  0.14;  up 0+00:05:16        16:42:02
top.120512164400:last pid:  1499;  load avg:  0.39,  0.35,  0.19;  up 0+00:07:15        16:44:01
top.120512164600:last pid:  1513;  load avg:  0.10,  0.26,  0.17;  up 0+00:09:16        16:46:02

Or grep Memory:

top.120512064600:Memory: 16G phys mem, 2031M free mem, 2048M total swap, 2048M free swap
top.120512064800:Memory: 16G phys mem, 2047M free mem, 2048M total swap, 2048M free swap
top.120512065000:Memory: 16G phys mem, 1443M free mem, 2048M total swap, 2048M free swap
top.120512065200:Memory: 16G phys mem, 1313M free mem, 2048M total swap, 2048M free swap
top.120512065400:Memory: 16G phys mem, 892M free mem, 2048M total swap, 2048M free swap
top.120512065601:Memory: 16G phys mem, 418M free mem, 2048M total swap, 2048M free swap
top.120512065800:Memory: 16G phys mem, 294M free mem, 2048M total swap, 2044M free swap

// restart

top.120512163801:Memory: 16G phys mem, 14G free mem, 2048M total swap, 2048M free swap

or grep trap:

top.120512064800:Kernel: 50542 ctxsw, 13 trap, 113144 intr, 850 syscall, 9 flt    
top.120512065000:Kernel: 76357 ctxsw, 9 trap, 199203 intr, 399 syscall, 9 flt   
top.120512065200:Kernel: 72294 ctxsw, 13 trap, 254779 intr, 481 syscall, 9 flt    
top.120512065400:Kernel: 87671 ctxsw, 11 trap, 256663 intr, 401 syscall, 11 flt    
top.120512065601:Kernel: 72696 ctxsw, 11 trap, 281765 intr, 402 syscall, 11 flt    
top.120512065800:Kernel: 77316 ctxsw, 458 trap, 272329 intr, 412 syscall, 450 flt    
// restarted here    
top.120512163801:Kernel: 1570 ctxsw, 10 trap, 2380 intr, 1741 syscall, 9 flt

this one is from echo "::memstat" | mdb -k:

top.120512064800:ZFS File Data             2898132             11320   69%    
top.120512065000:ZFS File Data             3039466             11872   73%    
top.120512065200:ZFS File Data             3081508             12037   74%    
top.120512065400:ZFS File Data             3188175             12453   76%    
top.120512065601:ZFS File Data             3309405             12927   79%    
top.120512065800:ZFS File Data             3393392             13255   81%    
// restart    
top.120512163801:ZFS File Data               70094               273    2%    
top.120512164000:ZFS File Data               93547               365    2%    
top.120512164200:ZFS File Data              197571               771    5%    
top.120512164400:ZFS File Data             1175965              4593   28%    
top.120512164600:ZFS File Data             1205865              4710   29%    
top.120512164800:ZFS File Data             2537072              9910   61%

The ZFS pool is not corrupted, the actual load is below the average (compared to our other file servers), the hardware seems to be ok, too.

What do you think what might be the reason for this behavior? What other statistics do I need to collect?

Edit:

$ zpool status -v
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0h6m with 0 errors on Wed Apr 25 14:40:49 2012
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c3t0d0s0  ONLINE       0     0     0
            c3t1d0s0  ONLINE       0     0     0

errors: No known data errors

  pool: volume
 state: ONLINE
  scan: resilvered 285G in 2h57m with 0 errors on Mon May  7 22:01:38 2012
config:

        NAME                                       STATE     READ WRITE CKSUM
        volume                                     ONLINE       0     0     0
          raidz1-0                                 ONLINE       0     0     0
            c0t600C0FF00012FBB1F749674F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7DDDA1154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB1DEA1154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D7CA2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB1EAA1154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7DEBA1154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB1F0A1154F01000000d0  ONLINE       0     0     0
          raidz1-1                                 ONLINE       0     0     0
            c0t600C0FF00012FC7DFCA1154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB1FDA1154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D08A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB109A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D14A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB115A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D20A2154F01000000d0  ONLINE       0     0     0
          raidz1-2                                 ONLINE       0     0     0
            c0t600C0FF00012FBB171A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D2CA2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB12DA2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D38A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB139A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D44A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7DA3CA754F01000000d0  ONLINE       0     0     0
          raidz1-3                                 ONLINE       0     0     0
            c0t600C0FF00012FC7D50A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB151A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D5CA2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB15DA2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D68A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FBB169A2154F01000000d0  ONLINE       0     0     0
            c0t600C0FF00012FC7D70A2154F01000000d0  ONLINE       0     0     0
        spares
          c0t600C0FF00012FBB1D7A1154F01000000d0    AVAIL   
          c0t600C0FF000131E9277AD154F01000000d0    AVAIL   

errors: No known data errors

$ zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool                     24.4G   249G  39.5K  /rpool
rpool/ROOT                14.1G   249G    31K  legacy
rpool/ROOT/solaris        5.59M   249G  11.5G  /
rpool/ROOT/solaris-1      14.1G   249G  11.5G  /
rpool/ROOT/solaris-1/var  2.15G   249G  1.94G  /var
rpool/ROOT/solaris/var    2.71M   249G  1.29G  /var
rpool/dump                8.24G   250G  7.98G  -
rpool/export                63K   249G    32K  /export
rpool/export/home           31K   249G    31K  /export/home
rpool/swap                2.06G   249G  2.00G  -
volume                    8.77T  33.6T  6.77T  /volume
volume/gluster            33.5G  1.97T  33.5G  /volume/gluster

Edit 2

Here is a diff of various statistics: http://diffchecker.com/k19ZP458 (left: "normal" system state, right: just a minute before the crash)

I have a pool of 28 2TB-disks (56T) in 4 arrays of 7 disks. Since it's raidz1 (~RAID5), I'd expect 1 disk to be used for parity in each array, so the resulting volume should be 2TB*4*(7-1)=48TB, right?

Now, what I see on my system:

$ zpool list volume
NAME     SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
volume  50.5T   308K  50.5T   0%  1.00x  ONLINE  -

$ zfs list volume
NAME     USED  AVAIL  REFER  MOUNTPOINT
volume  2.00T  40.3T  75.8K  /volume

$ df -h /volume
Filesystem             Size   Used  Available Capacity  Mounted on
volume                  42T    75K        40T     1%    /volume

So, there are only 42T instead of 48T. Where are the missing 6TB? And where does the number 50.5T come from?