Alex's questions

I have a mixed Ubuntu 16.04/Ubuntu 18.04 based infrastructure and would like to set up and configure firewalld on hosts. I am going to use an Ansible role for that (I have a Vagrant-based setup to test the Ansible role).

When I run Ansible playbooks a freshly installed firewalld blocks incoming connections by default although I never specified this explicitly:

[ 2161.574030] FINAL_REJECT: IN=enp0s8 OUT= MAC=08:00:27:f9:00:46:0a:00:27:00:00:00:08:00 SRC=172.16.137.1 DST=172.16.137.182 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=36677 DF PROTO=TCP SPT=35098 DPT=455 WINDOW=29200 RES=0x00 SYN URGP=0
[ 4717.718425] FINAL_REJECT: IN=enp0s8 OUT= MAC=08:00:27:f9:00:46:0a:00:27:00:00:00:08:00 SRC=172.16.137.1 DST=172.16.137.182 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=55673 DF PROTO=TCP SPT=47624 DPT=456 WINDOW=29200 RES=0x00 SYN URGP=0
[ 4719.682792] FINAL_REJECT: IN=enp0s8 OUT= MAC=08:00:27:f9:00:46:0a:00:27:00:00:00:08:00 SRC=172.16.137.1 DST=172.16.137.182 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=60255 DF PROTO=TCP SPT=43836 DPT=457 WINDOW=29200 RES=0x00 SYN URGP=0
[ 4721.358343] FINAL_REJECT: IN=enp0s8 OUT= MAC=08:00:27:f9:00:46:0a:00:27:00:00:00:08:00 SRC=172.16.137.1 DST=172.16.137.182 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=42733 DF PROTO=TCP SPT=51850 DPT=458 WINDOW=29200 RES=0x00 SYN URGP=0

Is it possible somehow to start firewalld in a permissive mode then configure everything and then apply the configuration in one shot?

We have a MongoDB replication set consisting of three members:

        "members" : [
                {
                        "_id" : 6,
                        "host" : "10.0.0.17:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 2,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 7,
                        "host" : "10.0.0.18:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 2,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 8,
                        "host" : "10.0.0.19:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 2,
                        "tags" : {
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],

The cluster is under moderate load, not more than few tens of requests per second. db.serverStatus() on a primary reports that almost all transactions are rolled back:

"transaction begins" : 2625009877,
"transaction checkpoint currently running" : 0,
"transaction checkpoint generation" : 22618,
"transaction checkpoint max time (msecs)" : 5849,
"transaction checkpoint min time (msecs)" : 153,
"transaction checkpoint most recent time (msecs)" : 1869,
"transaction checkpoint scrub dirty target" : 0,
"transaction checkpoint scrub time (msecs)" : 0,
"transaction checkpoint total time (msecs)" : 11017082,
"transaction checkpoints" : 22617,
"transaction checkpoints skipped because database was clean" : 0,
"transaction failures due to cache overflow" : 0,
"transaction fsync calls for checkpoint after allocating the transaction ID" : 22617,
"transaction fsync duration for checkpoint after allocating the transaction ID (usecs)" : 354402,
"transaction range of IDs currently pinned" : 0,
"transaction range of IDs currently pinned by a checkpoint" : 0,
"transaction range of IDs currently pinned by named snapshots" : 0,
"transaction range of timestamps currently pinned" : 8589934583,
"transaction range of timestamps pinned by the oldest timestamp" : 8589934583,
"transaction sync calls" : 0,
"transactions committed" : 30213144,
"transactions rolled back" : 2594972913,
"update conflicts" : 578

Basically, my questions are: What is going on here? Is it normal to have so many transactions and so many rollbacks? If not, what is a root cause and hot to fix it?

Upd.: We upgraded to 3.6.8-2.0 (this was the latest Percona package in 3.6 series) and the problem persisted.

We have a working FreeIPA installation, it's in production since February. Almost everything works as expected but when we try to run command-line FreeIPA-related tools none of them work:

[admin@ipa ~]$ kinit admin
Password for [email protected]: 
[admin@ipa ~]$ klist
Ticket cache: KEYRING:persistent:8800000
Default principal: [email protected]

Valid starting       Expires              Service principal
06/30/2014 21:19:30  07/01/2014 21:19:12  krbtgt/[email protected]
[admin@ipa ~]$ ipa pwpolicy-show global_policy
ipa: ERROR: Kerberos error: ('Unspecified GSS failure.  Minor code may provide more information', 851968)/('No Kerberos credentials available', -1765328243)
[admin@ipa ~]$

I'm not a Kerberos expert and don't really know what to check. How can we debug and resolve this?

Update: when I add -vv I get the following:

[admin@ipa ~]$ ipa -vv pwpolicy-show global_policy
ipa: INFO: trying https://ipa.example.com/ipa/xml
ipa: INFO: Forwarding 'pwpolicy_show' to server 'https://ipa.example.com/ipa/xml'
ipa: ERROR: Kerberos error: ('Unspecified GSS failure.  Minor code may provide more information', 851968)/('No Kerberos credentials available', -1765328243)
[admin@ipa ~]$

Update 2: the content of /etc/krb5.conf follows:

includedir /var/lib/sss/pubconf/krb5.include.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = true
 rdns = false
 ticket_lifetime = 24h
 forwardable = yes
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 EXAMPLE.COM = {
  kdc = ipa.example.com:88
  master_kdc = ipa.example.com:88
  admin_server = ipa.example.com:749
  default_domain = example.com
  pkinit_anchors = FILE:/etc/ipa/ca.crt
}

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM

[dbmodules]
  EXAMPLE.COM = {
    db_library = ipadb.so
  }

Update 3: This is a single-server installation, the distro is Fedora 19 and FreeIPA version is 3.3.5

We have a FreeIPA-based system, admin's password has expired and needs to be changed but the standard password changing procedure over SSH fails:

sashka@cellar ~ ssh [email protected]
[email protected]'s password: 
Password expired. Change your password now.
Last failed login: Mon Jun 30 15:38:21 MSK 2014 from 116.10.191.195 on ssh:notty
There were 6071 failed login attempts since the last successful login.
Last login: Wed Apr 16 19:28:54 2014
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for user admin.
Current Password: 
New password: 
Retype new password: 
Password change failed. Server message: Current password's minimum life has not expired

Password not changed.
passwd: Authentication token manipulation error
Connection to ipa.xxxxxxxxxx.com closed.

If we try to change the password using passwd it fails too with the same error message:

[admin@ipa ~]$ passwd
Changing password for user admin.
Current Password: 
New password: 
Retype new password: 
Password change failed. Server message: Current password's minimum life has not expired

Password not changed.
passwd: Authentication token manipulation error
[admin@ipa ~]$

What should we do to resolve this situation?

I have a CentOS-based host and a KVM Debian-based virtual machine on it. A host has an Ethernet bridge on its external network interface, this bridge is used by KVM:

br0       Link encap:Ethernet  HWaddr 00:25:90:01:5E:92  
          inet addr:5.XX.XX.84  Bcast:5.XX.XX.255  Mask:255.255.255.0
          inet6 addr: fe80::fc54:ff:feaf:95b3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2893439068 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2943859744 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3109906781642 (2.8 TiB)  TX bytes:3271403241664 (2.9 TiB)

br0:0     Link encap:Ethernet  HWaddr 00:25:90:01:5E:92  
          inet addr:10.228.0.1  Bcast:10.228.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

The bridge has two IPs, one external and another one from a virtual LAN between the host and the guest. It acts as a default gateway for the guest. STP is turned off on the bridge.

The problem is that the guest got an odd routing rule somehow:

root@new:~# ip route get 50.31.164.148
50.31.164.148 via 5.XX.XX.81 dev eth0  src 10.228.0.250 
    cache  ipid 0x0dfb rtt 4.781s rttvar 4.297s ssthresh 7 cwnd 9
root@new:~# 

5.XX.XX.81 is a default gateway of the host and I can't find this IP anywhere in static routing tables on the guest:

root@new:~# ip route list
default via 10.228.0.1 dev eth0 
10.116.0.0/16 via 10.116.0.146 dev tun0 
10.116.0.146 dev tun0  proto kernel  scope link  src 10.116.0.145 
10.228.0.0/24 dev eth0  proto kernel  scope link  src 10.228.0.250 

I wonder how this was possible at all and what should we do to prevent situations like this? Of course, ip route flush cache saved us but we definitely want to eliminate problem itself not to blindly flush the routing cache periodically.

We have a 100G ZVOL on a FreeBSD 10.0-CURRENT host which claims to use 176G of disk space:

root@storage01:~ # zfs get all zroot/DATA/vtest
NAME              PROPERTY              VALUE                  SOURCE
zroot/DATA/vtest  type                  volume                 -
zroot/DATA/vtest  creation              Fri May 24 20:44 2013  -
zroot/DATA/vtest  used                  176G                   -
zroot/DATA/vtest  available             10.4T                  -
zroot/DATA/vtest  referenced            176G                   -
zroot/DATA/vtest  compressratio         1.00x                  -
zroot/DATA/vtest  reservation           none                   default
zroot/DATA/vtest  volsize               100G                   local
zroot/DATA/vtest  volblocksize          8K                     -
zroot/DATA/vtest  checksum              fletcher4              inherited from zroot
zroot/DATA/vtest  compression           off                    default
zroot/DATA/vtest  readonly              off                    default
zroot/DATA/vtest  copies                1                      default
zroot/DATA/vtest  refreservation        none                   local
zroot/DATA/vtest  primarycache          all                    default
zroot/DATA/vtest  secondarycache        all                    default
zroot/DATA/vtest  usedbysnapshots       0                      -
zroot/DATA/vtest  usedbydataset         176G                   -
zroot/DATA/vtest  usedbychildren        0                      -
zroot/DATA/vtest  usedbyrefreservation  0                      -
zroot/DATA/vtest  logbias               latency                default
zroot/DATA/vtest  dedup                 off                    default
zroot/DATA/vtest  mlslabel                                     -
zroot/DATA/vtest  sync                  standard               default
zroot/DATA/vtest  refcompressratio      1.00x                  -
zroot/DATA/vtest  written               176G                   -
zroot/DATA/vtest  logicalused           87.2G                  -
zroot/DATA/vtest  logicalreferenced     87.2G                  -
root@storage01:~ # 

This looks like a bug, how can it consume more than its volsize if it does not have snapshots, reservations and children? Or maybe we are missing something?

Upd:

Results of zpool status -v:

root@storage01:~ # zpool status -v
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h6m with 0 errors on Thu May 30 05:45:11 2013
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          raidz2-0     ONLINE       0     0     0
            gpt/disk0  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
            gpt/disk2  ONLINE       0     0     0
            gpt/disk3  ONLINE       0     0     0
            gpt/disk4  ONLINE       0     0     0
            gpt/disk5  ONLINE       0     0     0
        cache
          ada0s2       ONLINE       0     0     0

errors: No known data errors
root@storage01:~ # 

Results of zpool list:

root@storage01:~ # zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  16.2T   288G  16.0T     1%  1.05x  ONLINE  -
root@storage01:~ # 

Results of zfs list:

root@storage01:~ # zfs list
NAME                            USED  AVAIL  REFER  MOUNTPOINT
zroot                           237G  10.4T   288K  /
zroot/DATA                      227G  10.4T   352K  /DATA
zroot/DATA/NFS                  288K  10.4T   288K  /DATA/NFS
zroot/DATA/hv                  10.3G  10.4T   288K  /DATA/hv
zroot/DATA/hv/hv001            10.3G  10.4T   144K  -
zroot/DATA/test                 288K  10.4T   288K  /DATA/test
zroot/DATA/vimage              41.3G  10.4T   288K  /DATA/vimage
zroot/DATA/vimage/vimage_001   41.3G  10.5T  6.47G  -
zroot/DATA/vtest                176G  10.4T   176G  -
zroot/SYS                      9.78G  10.4T   288K  /SYS
zroot/SYS/ROOT                  854M  10.4T   854M  /
zroot/SYS/home                 3.67G  10.4T  3.67G  /home
zroot/SYS/tmp                   352K  10.4T   352K  /tmp
zroot/SYS/usr                  4.78G  10.4T   427M  /usr
zroot/SYS/usr/local             288K  10.4T   288K  /usr/local
zroot/SYS/usr/obj              3.50G  10.4T  3.50G  /usr/obj
zroot/SYS/usr/ports             895K  10.4T   320K  /usr/ports
zroot/SYS/usr/ports/distfiles   288K  10.4T   288K  /usr/ports/distfiles
zroot/SYS/usr/ports/packages    288K  10.4T   288K  /usr/ports/packages
zroot/SYS/usr/src               887M  10.4T   887M  /usr/src
zroot/SYS/var                   511M  10.4T  1.78M  /var
zroot/SYS/var/crash             505M  10.4T   505M  /var/crash
zroot/SYS/var/db               1.71M  10.4T  1.43M  /var/db
zroot/SYS/var/db/pkg            288K  10.4T   288K  /var/db/pkg
zroot/SYS/var/empty             288K  10.4T   288K  /var/empty
zroot/SYS/var/log               647K  10.4T   647K  /var/log
zroot/SYS/var/mail              296K  10.4T   296K  /var/mail
zroot/SYS/var/run               448K  10.4T   448K  /var/run
zroot/SYS/var/tmp               304K  10.4T   304K  /var/tmp
root@storage01:~ # 

Upd 2:

We created a number of ZVOLs with different parameters and used dd to move the content. We noticed another odd thing, disk usage was normal for ZVOLs with 16k and 128k volblocksize and it remained abnormal for a ZVOL with 8k volblocksize even after dd (so this is not a fragmentation issue):

root@storage01:~ # zfs get all zroot/DATA/vtest-3
NAME                PROPERTY              VALUE                  SOURCE
zroot/DATA/vtest-3  type                  volume                 -
zroot/DATA/vtest-3  creation              Fri May 31  7:35 2013  -
zroot/DATA/vtest-3  used                  201G                   -
zroot/DATA/vtest-3  available             10.2T                  -
zroot/DATA/vtest-3  referenced            201G                   -
zroot/DATA/vtest-3  compressratio         1.00x                  -
zroot/DATA/vtest-3  reservation           none                   default
zroot/DATA/vtest-3  volsize               100G                   local
zroot/DATA/vtest-3  volblocksize          8K                     -
zroot/DATA/vtest-3  checksum              fletcher4              inherited from zroot
zroot/DATA/vtest-3  compression           off                    default
zroot/DATA/vtest-3  readonly              off                    default
zroot/DATA/vtest-3  copies                1                      default
zroot/DATA/vtest-3  refreservation        103G                   local
zroot/DATA/vtest-3  primarycache          all                    default
zroot/DATA/vtest-3  secondarycache        all                    default
zroot/DATA/vtest-3  usedbysnapshots       0                      -
zroot/DATA/vtest-3  usedbydataset         201G                   -
zroot/DATA/vtest-3  usedbychildren        0                      -
zroot/DATA/vtest-3  usedbyrefreservation  0                      -
zroot/DATA/vtest-3  logbias               latency                default
zroot/DATA/vtest-3  dedup                 off                    default
zroot/DATA/vtest-3  mlslabel                                     -
zroot/DATA/vtest-3  sync                  standard               default
zroot/DATA/vtest-3  refcompressratio      1.00x                  -
zroot/DATA/vtest-3  written               201G                   -
zroot/DATA/vtest-3  logicalused           100G                   -
zroot/DATA/vtest-3  logicalreferenced     100G                   -
root@storage01:~ # 

and

root@storage01:~ # zfs get all zroot/DATA/vtest-16
NAME                 PROPERTY              VALUE                  SOURCE
zroot/DATA/vtest-16  type                  volume                 -
zroot/DATA/vtest-16  creation              Fri May 31  8:03 2013  -
zroot/DATA/vtest-16  used                  102G                   -
zroot/DATA/vtest-16  available             10.2T                  -
zroot/DATA/vtest-16  referenced            101G                   -
zroot/DATA/vtest-16  compressratio         1.00x                  -
zroot/DATA/vtest-16  reservation           none                   default
zroot/DATA/vtest-16  volsize               100G                   local
zroot/DATA/vtest-16  volblocksize          16K                    -
zroot/DATA/vtest-16  checksum              fletcher4              inherited from zroot
zroot/DATA/vtest-16  compression           off                    default
zroot/DATA/vtest-16  readonly              off                    default
zroot/DATA/vtest-16  copies                1                      default
zroot/DATA/vtest-16  refreservation        102G                   local
zroot/DATA/vtest-16  primarycache          all                    default
zroot/DATA/vtest-16  secondarycache        all                    default
zroot/DATA/vtest-16  usedbysnapshots       0                      -
zroot/DATA/vtest-16  usedbydataset         101G                   -
zroot/DATA/vtest-16  usedbychildren        0                      -
zroot/DATA/vtest-16  usedbyrefreservation  886M                   -
zroot/DATA/vtest-16  logbias               latency                default
zroot/DATA/vtest-16  dedup                 off                    default
zroot/DATA/vtest-16  mlslabel                                     -
zroot/DATA/vtest-16  sync                  standard               default
zroot/DATA/vtest-16  refcompressratio      1.00x                  -
zroot/DATA/vtest-16  written               101G                   -
zroot/DATA/vtest-16  logicalused           100G                   -
zroot/DATA/vtest-16  logicalreferenced     100G                   -
root@storage01:~ # 

We are using stock HBase 0.94.4 on Hadoop 1.0.4. One of HBase regions stuck in transition state and I got the following when I run /opt/hbase/bin/hbase hbck:

ERROR: Region { meta => dev1_sliceagg_location_file,,1369128923119.21accc8b27bbd501ed4d3575d6ee725e., hdfs => hdfs://192.168.3.100:8020/hbase/dev1_sliceagg_location_file/21accc8b27bbd501ed4d3575d6ee725e, deployed =>  } not deployed on any region server.
ERROR: Region { meta => crash_experiment_sliceagg_client_file,,1369316587953.46e475f415d83f0d5caebccf67acc696., hdfs => hdfs://192.168.3.100:8020/hbase/crash_experiment_sliceagg_client_file/46e475f415d83f0d5caebccf67acc696, deployed =>  } not deployed on any region server.
ERROR: Region { meta => dev1_sliceagg_client_file,\x94\xDC\x97\x85\x94\x15\xAFO\xFEv\xE5}2\xBA\xE6\xC5\x8E\x87'\x0CG\x04\xCF)Q\xE1\xE7\x82\x0Dl\x8A+\x90\x18\xF8{2?\xD2]~6oO\x0F\\x97\x96\xBF\xE5Fc6|\xE8x\xF6+\x09s\xAF\xC9\xC3\xC8\x00<\x11\x00\x00\x00\x00\x00,1369315360949.92fc7ad4623318547cf7f4cb13e3afdc., hdfs => hdfs://192.168.3.100:8020/hbase/dev1_sliceagg_client_file/92fc7ad4623318547cf7f4cb13e3afdc, deployed =>  } not deployed on any region server.
13/05/23 18:54:16 DEBUG util.HBaseFsck: There are 64 region info entries
ERROR: There is a hole in the region chain between \x94\xDC\x97\x85\x94\x15\xAFO\xFEv\xE5}2\xBA\xE6\xC5\x8E\x87'\x0CG\x04\xCF)Q\xE1\xE7\x82\x0Dl\x8A+\x90\x18\xF8{2?\xD2]~6oO\x0F\\x97\x96\xBF\xE5Fc6|\xE8x\xF6+\x09s\xAF\xC9\xC3\xC8\x00<\x11\x00\x00\x00\x00\x00 and \xC80\xCD\x96\xBF-\xB0\xB6hm\x80\xE5\xD7\xDE\xAF\xB0\x0ANWW\xAE\x09\xFA\x96"\xE3\x15\x8C\xC1\xAE\xF1\x14\xEDWNB\x0EW7N2\x8C|Re\x04\xEC\xA5i\xC1d(yf\xF0`\x19\xEC |\xB1\x7F,T@6\x00\x00\x00\x00\x00\x00.  You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table dev1_sliceagg_client_file
ERROR: (region dev1_sliceagg_location_file,\x80+\x02)\xD9\x04\xE2\x8C\x1E\xA9\xA5'J\xB4W\xFC\xD4\x8C\x86Kgx\x87"\x0C\x14\x8F\xCD\x00p\x11\xEB\xB7;\x98\x9B02J[\x07\xF0\xE8\xAE\xC1m\xFF\xA4\x00$\x01\x00\x00\x00\x00\x00\x00\x00\x03\xEE\x00\x00\x00\x00\x00\x00?\xB2\x00\x00\x00\x00\x00\x00\x0A\xB5,1369128923119.f7b1c0288f9fcc36ebceca091103ac18.) First region should start with an empty key.  You need to  create a new region and regioninfo in HDFS to plug the hole.
ERROR: Found inconsistency in table dev1_sliceagg_location_file
13/05/23 18:54:17 WARN regionserver.StoreFile: Failed match of store file name hdfs://192.168.3.100:8020/hbase/crash_experiment_sliceagg_file_stat/06f163c5f5e79b02e260f3b2752c9cb8/.oldlogs/hlog.1369315359473
13/05/23 18:54:17 WARN regionserver.StoreFile: Failed match of store file name hdfs://192.168.3.100:8020/hbase/-ROOT-/70236052/.oldlogs/hlog.1358951260249
13/05/23 18:54:17 WARN regionserver.StoreFile: Failed match of store file name hdfs://192.168.3.100:8020/hbase/dev1_sliceagg_client_file/92fc7ad4623318547cf7f4cb13e3afdc/.oldlogs/hlog.1369315360956
13/05/23 18:54:17 WARN regionserver.StoreFile: Failed match of store file name hdfs://192.168.3.100:8020/hbase/crash_experiment_sliceagg_client_file/46e475f415d83f0d5caebccf67acc696/.oldlogs/hlog.1369316587995
13/05/23 18:54:17 WARN regionserver.StoreFile: Failed match of store file name hdfs://192.168.3.100:8020/hbase/.META./1028785192/.oldlogs/hlog.1358951260483

/opt/hbase/bin/hbase hbck -fix does not fix anything becuase it gets stuck printing Region still in transition, waiting for it to become assigned error message. /opt/hbase/bin/hbase hbck -repairHoles does not help too. What should we do to resolve this situation?

We have an NFS share on a Debian Linux host which is mounted on a Win XP host using SFU. We can create folders on a share or delete files from it but it's not possible to perform file write or read operations on it (say, copy file.ext n:\ blocks for a long time and finally terminates with The remote system refused the network connection error). We tried both TCP and UDP on a client side when mounting. All ports are open on firewall between these two machines. There is nothing relevant in Win XP Event Log and in syslog on a server side. And this is not a permission problem obviously because we can create folders. What can we do to find a root cause of this?

Update: I captured an NFS session using Wireshark and found that the root cause was a locking problem, the NFS server keeped answering NFS_DENIED_GRACE_PERIOD status to client calls. And the question now is how to fix this locking issue?

We run a simple deployment script remotely using a command like ssh [email protected] sudo /root/run-chef-client.sh. It started to hang today because sshd waited forever on the 10.170.4.11 even after sudo had finished already. We started sshd in debug mode and got two different kind of logs. The following is a normal log when the session does not hang:

debug1: Received SIGCHLD.
debug1: session_by_pid: pid 23187
debug1: session_exit_message: session 0 channel 0 pid 23187
debug1: session_exit_message: release channel 0
Received disconnect from 10.170.4.6: 11: disconnected by user

And when it hangs we get the following:

debug1: Received SIGCHLD.
debug1: session_by_pid: pid 24209
debug1: session_exit_message: session 0 channel 0 pid 24209
debug1: session_exit_message: release channel 0

Our understanding is that the server process waits for some communication from a client side and never gets it. It's hard to tell if it is a client side or a server side problem. We tried to run sshd under strace but did not succeed because a SUID bit on sudo was ignored it this case. So, what else should we try to debug/prevent this situations?

The book "HBase: The definitive guide" states that

Installing different filesystems on a single server is not recommended. This can have adverse effects on performance as the kernel may have to split buffer caches to support the different filesystems. It has been reported that, for certain operating systems, this can have a devastating performance impact.

Does this really apply to Linux? I have never seen the buffer cache bigger than 300 Mbytes and most modern servers have gigabytes of RAM so splitting the buffer cache between different filesystems should not be an issue. Am I missing something else?

My customer has quite a large (the total "data" folder size is 200G) PostgreSQL database and we are working on a disaster recovery plan. We have identified three different types of disasters so far: hardware outage, too much load and unintentional data loss due to erroneously executed bad migration (like DELETE or ALTER TABLE DROP COLUMN).

First two types seem to be easy to mitigate but we can't elaborate a good mitigation plan for the third type. I proposed to use ZFS and frequent (hourly) snapshots but "ZFS" means "OpenIndiana" these days and our Ops engineers do not have much expertise in it, so using OpenIndiana imposes another risk. Colleagues try to convince me that restoring from PostgreSQL PITR backup can be as fast as restoring from a ZFS snapshot but I highly doubt that replaying, say, 50G of archived WALs can be considered "fast".

What other options are we missing? Is ZFS an only viable alternative? Can we get a fast Pg DB restore time in the Linux environment?

I have a web Linux-based infrastructure which consists of 15 virtual machines and over 50 various services. It is fully controlled by Chef. Most of the services are developed internally.

Basically the current deployment process is triggered by a shell script. A build system (a mix of Python and shell scripts) packages the services as .deb files and puts these packages into a repo. It runs apt-get update on all 15 nodes then because the standard Chef apt cookbook only runs apt-get once per day and we definitely do not want to run apt-get update unconditionally on each chef-client wake. The build system restarts chef-client daemons on all 15 nodes finally (we need this step because of pull Chef nature).

The current process has a number of drawbacks we want to address. First off, it is asynchronous because the deployment script does not check chef-client logs after restart so we don't even know if the deployment was successful. It does not even wait for Chef clients to complete the cycle. Second, we definitely do not want to force chef-client restarts on all nodes because we usually deploy only a small number of packages. And third, I am not quite sure using chef-client for deployment is legitimate, probably we are just doing it wrong from the start. Please share your thoughts/experience.

I use Debian 6.0.3 x86_64 and MySQL 5.5.20-1~dotdeb.0-log from the Dotdeb repository. MySQL process started to consume a lot of "sy" CPU time serveral hours ago according to this graph. I was not able to connect to running mysqld process and had to kill it. I found nothing useful in logs. My setup seems to be quite common (I assume Dotdeb just redistributes stock MySQL versions) and I have never seen anything like this before. What is the possible root cause of this? How can I prevent this situation in the future?

My system is a Debian 6.0.3 x86_64 box w/kernel 2.6.32-5-openvz-amd64, it hosts a number of OpenVZ containers. I have recently migrated from LXC/newer kernel from backports repo to the stable OpenVZ kernel to resolve network issues but got another issue - the CPU spends a lot of time processing software interrupts. Here is a Munin graph for CPU time: http://prntscr.com/arjzl.

I added nohz=off and highres=off to the kernel command line but that did not help much. And, the number of interrupts is quite low on this box according to another Munin graph: http://prntscr.com/ark19, so the interrupts seem to be not the cause of this issue. When I run top, zabbix_server is the most CPU consumer but I don't see any relation between zabbix_server and software interrupts.

I'm not sure how to proceed with investigation, this looks like a kernel bug or a kernel module bug to me, but I don't know hot to track this down to the guilty process/module. Maybe there is another kernel command line parameter to tune. What should I try next?

We are running a Ruby on Rails web app under Unicorn. Our app is not strictly CPU bound (we have a dual Xeon E5645 system w/12 cores and a peak load average value is around 6). We started with 40 Unicorn workers initially but application memory footprint increased over time. So, now we have to lower the number of worker processes. I thought that the standard (number of CPU cores + 1) formula applies to Unicorn too but my colleague tried to convince me we should reserve more Unicorn instances per CPU and provided this link. Yet, I am not exactly sure why do we need to spend so much memory on idle Unicorn processes.

My question is: what is the reason to have more than one Unicorn instance per CPU core? Is it due to some architectural peculiarity of Unicorn? I am aware that busy Unicorn processes can't accept new connections (we are using UNIX domain sockets to communicate to Unicorn instances BTW) but I thought backlog was introduced exactly to address this. Is it possible to overcome this 2 to 8 Unicorn instances per CPU rule anyhow?

I am setting up a Debian-based host with a bunch of KVM-based virtual machines. Images of these VMs reside on a ZFS volume (I use ZFS-FUSE 0.6.9). Performance results seem to be very different from my another system, based on CentOS 6. I am concerned that Debian implementation of ZFS-FUSE ignores settings from /etc/zfs/zfsrc. How can I monitor actual state of ZFS-FUSE, say, ARC cache usage, vdev caches usage, etc?

My system is CentOS 6 x86_64 with root partition formatted as ext4. df reports around 3Gb as used space:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md1              20158260   3433724  15700540  18% /

but du -sm -x / claims less than a single Gb is used actually:

[root@xxxx ~]# du -sm -x /
948     /

I wonder what is going on here. Usage numbers changed right after reboot. The filesystem claims it's clean, no errors in logs. I found this, but it did not explain a root of the problem. Should I just reformat the partition? Is there any way to track down this extra usage?

I also did the following to check that I don't have any data hidden from du by mounts on top of non-empty mountpoints:

[root@xxxx ~]# mount -o bind / /mnt/root
[root@xxxx ~]# du -sm /mnt/root/
949     /mnt/root/
[root@xxxx ~]#

No, that's not my case.

Our SMTP provider requires to distribute emails evenly between four different hosts. I understand that the standard way to distribute this kind of load is utilizing DNS round robin feature, but the problem is that provider's mailservers should be addressed using symbolic names not IPs. What is the best way to handle this? Will setting multiple CNAME records work or should I use any internal Sendmail/Postfix/Exim/etc balancing capability (which I am currently not aware of)?

I have a brand new server w/48G RAM which I am going to use as a DB server. I don't expect problems with disk reads, but I definitely would like to optimize for write intensive load patterns.

The OS is RHEL 5.6 and the FS is ext3, I've added "noatime" and "data=writeback" to /etc/fstab already and the latter option helped to reduce LA a lot. My next goal is to optimize the pdflush process wherever possible. I tried to apply tweaks mentioned here, but to no avail. Probably this information is just outdated.

What are my further options? Should I continue experimenting with pdflush or maybe it's better just to leave it as is? I tend to lower dirty_ratio and dirty_background_ratio sysctl values to increase I/O smoothness but these values seem to be not related to performance, the load pattern in Munin under stress test is basically the same.

Should I also try different I/O scheduler? Can I benefit from having lots of RAM in a write intensive setup at all? I understand that disk I/O speed and latency are nothing to do with the RAM, but my goal is not to magically write to disk faster, but to increase system stability and to implement some kind of graceful degradation.

Let's say I have good backups and can accept further data consistency tradeoffs like "data=writeback".
Thank you.

I have a MySQL (actually, MariaDB 5.2) setup deployed on a Rackspacecloud 4Gb node. All tables are InnoDB, InnoDB buffer pool size is 2G and it is 95% full most of the time. I use ext3 as a filesystem (not sure if I can use anything else on a cloud node). My application does a lot of writes to two statistics tables. Both tables have an auto_increment PK and an unique index. These tables are cleaned every 10 mins by a cron job. This cron job also does a number of long-running queries to process raw stats.

The problem is that I get high I/O spikes from time to time. It looks like they are related to a number of uncheckpointed bytes in the InnoDB transaction log. The average number of uncheckpointed bytes is 18.6M but it gets to 80-90M when I see these spikes. Not sure what causes what here.

Iotop shows that kjournald is the top performer when these spikes take place. I remounted the FS with "data=writeback" option but to no avail, kjournald was still on the top. I also tried to decrease swappiness and to increase queue/nr_requests for the root device, but that did not help either.

I am not sure what to do next. I wonder why kjournald generates such a huge I/O load at all on a FS with only metadata being journaled. Should I try to tune a commit interval? Or probably to use ionice?