Nikita Kipriyanov's questions

I have a server which runs Proxmox VE 8.2 (based on Debian 12), fully updated, which uses multipath SAN. There we have a space allocated, ten devices of 2T each, which are seen 16 times each (the number of paths) and which are mapped into 10 multipathed virtual devices and collected into a single LVM group, where a logical volume is carved. No problems on this part.

It was initialized "on the fly" and running all right, until the host was rebooted. After that I noticed that whenever one runs any LVM-related command (lvs, etc.) in the system shell, it emits the warning in the form:

WARNING: Device mismatch detected for <LV> which is accessing <list of devices like
 /dev/sdak, /dev/sdaz, /dev/sdbn, ...> instead of <list of the devices
 like /dev/mapper/oamdwhdg-01, /dev/mapper/oamdwhdg-02, ...>

These /dev/mapper/oamdwhdg-XX are in reality the multipath devices, they were named that way for operational reasons in /etc/multipath.conf:

multipaths {
...
    multipath {
        wwid 36...
        alias oamdwhdg-04
    }
...
}

I.e. for whatever reason, the LV is mapped using backend devices, instead of using multipath virtual devices.

So I updated the filter in /etc/lvm/lvm.conf, which now looks like this:

global_filter=[ "a|/dev/sda3|", "a|/dev/mapper/oamdwhdg|", "r|.*|" ]

(I updated it, not added, it was global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|"] before with the comment added by pve-manager to avoid scanning ZFS zvols and Ceph rbds. I believe my filter is much more strict than this one.)

/dev/sda3 is the local disk where PVE is installed and it contains the VG pve which doesn't depend on SAN, so it is the only /dev/sd... which is not using multipath and it was whitelisted.

I tested this filter using vgscan and it shows both groups to be found.

Then I executed update-initramfs and rebooted. During boot, server failed into emergency shell. When I hit Ctrl+D there, it, however, booted almost normally: the multipathed VG is seen but not activated (as if vgchange -an oamdwhdg was not yet run). But then I activated it by hand perfectly normally with the said command and it worked flawlessly.

I suspect this is because multipath isn't correctly initialized before LVM in the initramfs, so it tried to set up mappings using /dev/sdXX devices. But why it failed to emergency shell when I added the filter I don't understand.

Two (very related) questions here: why emergency shell a.k.a. what is wrong and how to make it work as expected?

I'm trying to add HPE StoreOnce 3640 Gen4 to the Nagios using SNMP v3 (other SNMP versions aren't supported in this device), and I'm stuck. It has the SW version 4.2.1-1942.21.

This is not the first time configuring SNMP v3; I'v done it already on Mikroitk devices, Eltex devices, Cisco devices and even (for testing) configured snmpd on the Linux machine. However, this one doesn't work.

As described in the HPE manual, I'm creating the SNMP user nagios, selected the mode "authentication and privacy" (which requires to specify two passwords), set the authentication mode "SHA" and password, and privacy mode "AES" and another password. Enabled the SNMP agent, and it says "enabled" in the web interface.

Then, from the neighbor machine I run:

snmpwalk  -v3 -u nagios -l authPriv -a SHA -A <authPW> -x AES -X <privPW> 192.168.221.150 1.3.6.1.4.1.11.2.36.1.1.5.1.12

(this OID is described as "Version number for this device's hardware" in the MIB) and nothing happens. It says: snmpwalk: Timeout. tcpdump shows that StoreOnce just doesn't answer. Obviously, I can't check whether it is listening, because it's UDP. Nmap with UDP scan says the port 161 is filtered (however, this is not the definitive test, due to UDP nature).

I am sure both passwords are correct (and I tried to swap them, just to be sure). I am sure there is no network firewall between the device and the server, they are in the same IP subnet. Also I've tried to create a ordinary "web interface" user in StoreOnce with the same name, just for the case. Running similar command against Linux SNMP agent on another neighbor machine works fine (of course, there is no such OID on Linux machine, I walked just through 1.3.6.1), so I am sure I use snmpwalk correctly.

Probably I am missing something in the StoreOnce configuration?

There is pretty supported action to move from Mnesia to SQL, a special command ejabberdctl export2sql (in older versions it used to be called export2odbc). Some setups may need that because Mnesia is limited to 2GiB and blah blah blah.

In my case there is really no need to have an external database. There are around 10-15 active users and no more than 30 of total users, and the size of the export from SQL database is around 200 kB.

Somebody in the past did that thing, exported a database into SQL (specifically, MySQL). How can I revert that change?

Can export_piefxis help me somehow? What it does?

Update 1. After some exploration I found that only two modules have some data in the SQL database: mod_roster and mod_vcard. So only those need to be converted. Is there a way to convert their data externally and load into ejabberd? I mean, there is a ejabberdctl dump_table, but again no reverse command.

Also, export_piefxis doesn't seem to help. It just exports zero content, while there is data in the database and it is fetched when I connect with e.g. newly instantiated client.

Update 2. Software: Debian 11.2 LXC container running on Proxmox VE 6.4-1 host. Ejabberd 21.01-2 installed with apt from standard Debian repo.

The database was migrated from old Ejabberd of around version 2.1, that was running on Gentoo and last updated in 2015. I don't have that machine at my discretion anymore. Mnesia database migration was done as follows:

# on the old machine
ejabberdctl backup /tmp/[email protected] 
ejabberdctl mnesia_change_nodename ejabberd@localhost ejabberd@jabber2 \
           /tmp/[email protected] /tmp/[email protected]
# on the new machine (starting with empty database)
ejabberdctl install_fallback /tmp/[email protected]
ejabberdctl stop
ejabberdctl start

The configuration on the new machine just references the same MySQL database for mod_roster and mod_vcard.

ejabberd.yml:

loglevel: info

log_rotate_count: 0

hosts:
  - "example.org"

host_config:
  "rterm.ru":
    sql_type: mysql
    sql_server: "dbserver"
    sql_database: "ejabberd"
    sql_username: "ejabberd"
    sql_password: "password"
    auth_method: [external]

certfiles:
  - "/etc/ejabberd/ejabberd.pem"

acme:
  auto: false

define_macro:
  'TLS_CIPHERS': "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
  'TLS_OPTIONS':
    - "no_sslv3"
    - "no_tlsv1"
    - "no_tlsv1_1"
    - "cipher_server_preference"
    - "no_compression"


c2s_ciphers: 'TLS_CIPHERS'
s2s_ciphers: 'TLS_CIPHERS'
c2s_protocol_options: 'TLS_OPTIONS'
s2s_protocol_options: 'TLS_OPTIONS'

listen:
  -
    port: 5222
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
    protocol_options: 'TLS_OPTIONS'
  -
    port: 5223
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    tls: true
    protocol_options: 'TLS_OPTIONS'
  -
    port: 5269
    ip: "::"
    module: ejabberd_s2s_in
    max_stanza_size: 524288
  -
    port: 5443
    ip: "::"
    module: ejabberd_http
    tls: true
    protocol_options: 'TLS_OPTIONS'
    request_handlers:
      /admin: ejabberd_web_admin
      /api: mod_http_api
      /bosh: mod_bosh
      /upload: mod_http_upload
      /ws: ejabberd_http_ws
  -
    port: 5280
    ip: "::"
    module: ejabberd_http
    tls: false
    request_handlers:
# This runs behind reverse HTTP proxy with Let's Encrypt, which is neighbour container on the same host, so this is safe
      /admin: ejabberd_web_admin
      /bosh: mod_bosh
      /upload: mod_http_upload
  -
    port: 1883
    ip: "::"
    module: mod_mqtt
    backlog: 1000

auth_method: [external]
extauth_program: "/etc/ejabberd/auth_mysql.py"
extauth_pool_size: 5

disable_sasl_mechanisms:
  - "X-OAUTH2"

s2s_use_starttls: required

acl:
  admin:
    user:
       - "[email protected]"

  local:
    user_regexp: ""
  loopback:
    ip:
      - 127.0.0.0/8
      - ::1/128

access_rules:
  local:
    allow: local
  c2s:
    deny: blocked
    allow: all
  announce:
    allow: admin
  configure:
    allow: admin
  muc_create:
    allow: local
  pubsub_createnode:
    allow: local
  trusted_network:
    allow: loopback

api_permissions:
  "console commands":
    from:
      - ejabberd_ctl
    who: all
    what: "*"
  "admin access":
    who:
      access:
        allow:
          - acl: loopback
          - acl: admin
      oauth:
        scope: "ejabberd:admin"
        access:
          allow:
            - acl: loopback
            - acl: admin
    what:
      - "*"
      - "!stop"
      - "!start"
  "public commands":
    who:
      ip: 127.0.0.1/8
    what:
      - status
      - connected_users_number

shaper:
  normal:
    rate: 3000
    burst_size: 20000
  fast: 200000

shaper_rules:
  max_user_sessions: 10
  max_user_offline_messages:
    5000: admin
    100: all
  c2s_shaper:
    none: admin
    normal: all
  s2s_shaper: fast

modules:
  mod_adhoc: {}
  mod_admin_extra: {}
  mod_announce:
    access: announce
  mod_avatar: {}
  mod_blocking: {}
  mod_bosh: {}
  mod_caps: {}
  mod_carboncopy: {}
  mod_client_state: {}
  mod_configure: {}
  mod_disco: {}
  mod_fail2ban: {}
  mod_http_api: {}
  mod_http_upload:
    put_url: https://@HOST@:5443/upload
    custom_headers:
      "Access-Control-Allow-Origin": "https://@HOST@"
      "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS"
      "Access-Control-Allow-Headers": "Content-Type"
  mod_last: {}
  mod_mqtt: {}
  mod_muc:
    access:
      - allow
    access_admin:
      - allow: admin
    access_create: muc_create
    access_persistent: muc_create
    access_mam:
      - allow
    default_room_options:
      mam: true
  mod_muc_admin: {}
  mod_offline:
    access_max_user_messages: max_user_offline_messages
  mod_ping: {}
  mod_pres_counter:
    count: 5
    interval: 60
  mod_privacy: {}
  mod_private: {}
  mod_proxy65:
    access: local
    max_connections: 5
  mod_pubsub:
    access_createnode: pubsub_createnode
    plugins:
      - flat
      - pep
    force_node_config:
      "eu.siacs.conversations.axolotl.*":
        access_model: open
      storage:bookmarks:
        access_model: whitelist
  mod_push: {}
  mod_push_keepalive: {}
  mod_roster:
    db_type: sql # This is what I want to get rid of
    versioning: true
  mod_s2s_dialback: {}
  mod_shared_roster: {}
  mod_sic: {}
  mod_stream_mgmt:
    resend_on_timeout: if_offline
  mod_stun_disco: {}
  mod_vcard:
    db_type: sql # And this
    search: false
  mod_vcard_xupdate: {}
  mod_version: {}

In DNS there was SRV record defined long ago, which allowed to direct some services for a domain to different machines. For example, this is how I publish a LDAP service for a domain example.com served on server1.example.net tcp port 389:

_ldap._tcp.example.com IN SRV 100 1 389 server1.example.net 

There may be more than one record of that kind and this way I can achieve some load balancing.

Support of corresponding DNS lookups is usual for some services, including LDAP, Kerberos, SIP, XMPP and some others. However, for email we use a "reduced form" of SRV record called MX record. It doesn't allow us to define weight and alter port (it always uses tcp 25). Some services don't use this type of lookup at all, for example, HTTP clients never issue corresponding DNS requests and always connect to the TCP port 80 (or 443 for HTTPS) of the corresponding host.

Recently a new draft appeared, which suggests the use of the special new type of DNS records, called SVCB record. It also defines a special type of SVCB record, a HTTPS record. In the draft authors confirm this is essentially a new type of SRV record.

So I'd like to understand why they invented a new type of a record with a whole new semantic, instead of pushing an adoption of well known and widely used SRV record? Why not use something like

_https._tcp.example.net IN SRV 100 1 443 server2.example.com

Disadvantages are similar: this needs change of a software. Old client software will neither do SRV nor SVCB/HTTPS DNS requests, so neither will work.

However, SVCB has a huge additional disadvantage: the record type is new and therefore it needs also DNS server updates and administrator training. By the way, a semantic of a record is not that simple.

As a professional administrator who has some dozen public DNS domains in charge, I am concerned with this.

What are significant advantages?

(the question was reformulated, I think it needed to be more structured)

We have a Proxmox VE on Dell PowerEdge R610 gen 8 system. The platform is old, but we use it for particular S/W which is well known having no benefits from modern CPU cores, but increases its performance linearly with CPU clock frequency, and 3.3GHz accomplishes the goal well. A performance analysis showed disk I/O is serous bottleneck, while others aren't.

HW config is:

• Dell PowerEdge R610 gen 8, BIOS v6.6.0 of 05/22/2018 (most recent), dual PSU - both seem to be OK. Server boots in UEFI.
• CPU: 2x Xeon X5680 (Nehalem, 12 cores total, 3.3GHz, boosts up to 3.6 GHz)
• RAM: 96 GiB - 6x Samsung M393B2K70DM0-YH9 (DDR3, 16GiB, 1333MT/s)
• Storage controller: LSI MegaRAID SAS 9240-4i, JBOD mode (SAS-MFI BIOS, FW v20.10.1-0107 - not the latest one)
• Storage: 2x new Samsung SSD 860 EVO 1TB, firmware RVT03B6Q

MegaRAID we use is not the build-in PERC. The built-in was only capable to do only 1.5 Gbit/S SATA which is way too slow, also JBOD or HBA mode are disabled. Unlike that, an added-on 9240-4i runs SSDs on their max interface speed of 6 Gbit/s, and allows for JBOD mode.

The card has no battery and no cache, so it was obvious it has too low performance when RAID was built with it, so both disks are configured as JBOD and used with software RAID. Theoretical maximum for 6 Gbit/s interface is 600 MB/s (considering 8-to-10-bits wire encoding), this is what to expect from a single drive sequential test.

We done extensive i/o tests both under Linux and under Windows, both with fio with same config. The only differences in the config were aio library (windowsaio in Windows, libaio in Linux) and test device specifications. fio config was adapted from this post: https://forum.proxmox.com/threads/pve-6-0-slow-ssd-raid1-performance-in-windows-vm.58559/#post-270657 . I can't show full fio outputs because this will hit ServerFault limit of 30k characters. I can share them somewhere else if somebody wants to see. Here I'll show only summary lines. Linux (Proxmox VE) was configured with MD RAID1 and "thick" LVM.

Caches inside SSDs are enabled:

# hdparm -W /dev/sd[ab]

/dev/sda:
 write-caching =  1 (on)

/dev/sdb:
 write-caching =  1 (on)

Devices run at full 6 Gb/s interface speed:

# smartctl -i /dev/sda
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.3.10-1-pve] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 860 EVO 1TB
Serial Number:    S4FMNE0MBxxxxxx
LU WWN Device Id: x xxxxxx xxxxxxxxx
Firmware Version: RVT03B6Q
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Feb  7 15:25:45 2020 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

# smartctl -i /dev/sdb
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.3.10-1-pve] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 860 EVO 1TB
Serial Number:    S4FMNE0MBxxxxxx
LU WWN Device Id: x xxxxxx xxxxxxxxx
Firmware Version: RVT03B6Q
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Feb  7 15:25:47 2020 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Partitions were carefully aligned to 1 MiB, and the "main" large partition which is LVM PV and where all tests were done starts exactly at 512 MiB:

# fdisk -l /dev/sd[ab]
Disk /dev/sda: 931,5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 860 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1DDCF7A0-D894-8C43-8975-C609D4C3C742

Device       Start        End    Sectors  Size Type
/dev/sda1     2048     524287     522240  255M EFI System
/dev/sda2   524288     526335       2048    1M BIOS boot
/dev/sda3   526336    1048575     522240  255M Linux RAID
/dev/sda4  1048576 1953525134 1952476559  931G Linux RAID


Disk /dev/sdb: 931,5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Samsung SSD 860 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 63217472-3D2E-9444-917C-4776100B2D87

Device       Start        End    Sectors  Size Type
/dev/sdb1     2048     524287     522240  255M EFI System
/dev/sdb2   524288     526335       2048    1M BIOS boot
/dev/sdb3   526336    1048575     522240  255M Linux RAID
/dev/sdb4  1048576 1953525134 1952476559  931G Linux RAID

There is no bitmap:

# cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md126 : active raid1 sda4[2] sdb4[0]
      976106176 blocks super 1.2 [2/2] [UU]

md127 : active raid1 sda3[2] sdb3[0]
      261056 blocks super 1.0 [2/2] [UU]

unused devices: <none>

LVM is created with 32 MiB PE size, so inside it everything is aligned to 32 MiB.

lsblk --discard shows no device supports any TRIM (even non-queued). This is probably because of LSI2008 chip does not know this command. Queued TRIM is blacklisted on these SSDs: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/ata/libata-core.c?id=9a9324d3969678d44b330e1230ad2c8ae67acf81 . Anyway, this is still the same Windows sees, so comparison is fair.

The I/O scheduler was "none" on both disks. I also tried "mq-deadline" (the default), it showed worse results in general.

Under that configuration, fio showed following results:

PVEHost-128K-Q32T1-Seq-Read  bw=515MiB/s (540MB/s), 515MiB/s-515MiB/s (540MB/s-540MB/s), io=97.5GiB (105GB), run=194047-194047msec 
PVEHost-128K-Q32T1-Seq-Write bw=239MiB/s (250MB/s), 239MiB/s-239MiB/s (250MB/s-250MB/s), io=97.7GiB (105GB), run=419273-419273msec
PVEHost-4K-Q8T8-Rand-Read    bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=799GiB (858GB), run=3089818-3089818msec
PVEHost-4K-Q8T8-Rand-Write   bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=799GiB (858GB), run=6214084-6214084msec
PVEHost-4K-Q32T1-Rand-Read   bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=98.7GiB (106GB), run=380721-380721msec
PVEHost-4K-Q32T1-Rand-Write  bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=99.4GiB (107GB), run=768521-768521msec
PVEHost-4K-Q1T1-Rand-Read    bw=16.8MiB/s (17.6MB/s), 16.8MiB/s-16.8MiB/s (17.6MB/s-17.6MB/s), io=99.9GiB (107GB), run=6102415-6102415msec
PVEHost-4K-Q1T1-Rand-Write   bw=36.4MiB/s (38.1MB/s), 36.4MiB/s-36.4MiB/s (38.1MB/s-38.1MB/s), io=99.8GiB (107GB), run=2811085-2811085msec

On the exactly same hardware configuration, Windows was configured with Logical Disk Manager mirroring. Results are:

WS2019-128K-Q32T1-Seq-Read  bw=1009MiB/s (1058MB/s), 1009MiB/s-1009MiB/s (1058MB/s-1058MB/s), io=100GiB (107GB), run=101535-101535msec
WS2019-128K-Q32T1-Seq-Write bw=473MiB/s (496MB/s), 473MiB/s-473MiB/s (496MB/s-496MB/s), io=97.8GiB (105GB), run=211768-211768msec
WS2019-4K-Q8T8-Rand-Read    bw=265MiB/s (278MB/s), 265MiB/s-265MiB/s (278MB/s-278MB/s), io=799GiB (858GB), run=3088236-3088236msec
WS2019-4K-Q8T8-Rand-Write   bw=130MiB/s (137MB/s), 130MiB/s-130MiB/s (137MB/s-137MB/s), io=799GiB (858GB), run=6272968-6272968msec
WS2019-4K-Q32T1-Rand-Read   bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=99.1GiB (106GB), run=536262-536262msec
WS2019-4K-Q32T1-Rand-Write  bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=99.4GiB (107GB), run=823544-823544msec
WS2019-4K-Q1T1-Rand-Read    bw=22.9MiB/s (24.0MB/s), 22.9MiB/s-22.9MiB/s (24.0MB/s-24.0MB/s), io=99.9GiB (107GB), run=4466576-4466576msec
WS2019-4K-Q1T1-Rand-Write   bw=41.4MiB/s (43.4MB/s), 41.4MiB/s-41.4MiB/s (43.4MB/s-43.4MB/s), io=99.8GiB (107GB), run=2466593-2466593msec

The comparsion:

windows   none     mq-deadline comment
1058MB/s  540MB/s  539MB/s     50% less than Windows, but this is expected
496MB/s   250MB/s  295MB/s     40-50% less than Windows!
278MB/s   278MB/s  278MB/s     same as Windows
137MB/s   138MB/s  127MB/s     almost same as Windows
198MB/s   278MB/s  276MB/s     40% more than Windows
130MB/s   139MB/s  130MB/s     similar to Windows
24.0MB/s  17.6MB/s 17.3MB/s    26% less than Windows
43.4MB/s  38.1MB/s 28.3MB/s    12-34% less than Windows

Linux MD RAID1 only reads from both drives if there are at least two threads. First test is single thread, so Linux will read from a single drive and will achieve a single drive performance. This is justifiable and this first test result is fine. But others...

These only host tests. When we compared what is going when we ran same tests inside VMs, the last lines showed even worse, in Windows VM under PVE (no ballooning fixed memory, fixed CPU frequency, virtio scsi v171, writeback with barriers), it displayed 70% less than under Windows under Hyper-V. Even Linux VM under PVE shows results much worse than Windows under Hyper-V:

                     windows, windows, linux,
                     hyper-v  pve      pve
128K-Q32T1-Seq-Read  1058MB/s 856MB/s  554MB/s
128K-Q32T1-Seq-Write 461MB/s  375MB/s  514MB/s
4K-Q8T8-Rand-Read    273MB/s  327MB/s  254MB/s
4K-Q8T8-Rand-Write   135MB/s  139MB/s  138MB/s
4K-Q32T1-Rand-Read   220MB/s  198MB/s  210MB/s
4K-Q32T1-Rand-Write  131MB/s  146MB/s  140MB/s
4K-Q1T1-Rand-Read    18.2MB/s 5452kB/s 8701kB/s
4K-Q1T1-Rand-Write   26.7MB/s 7772kB/s 10.7MB/s

During these tests, Windows under Hyper-V was quite responsible despite large I/O load, same Linux under PVE. But when Windows ran under PVE, its GUI was slow to crawl, RDP session tended to disconnect itself due to packet drop, and HA on the host was up to 48, which was mostly due to huge i/o wait!

During the test saw quite large load on a single core, which happened to serve a "megasas" interrupt. This card only shows a single interrupt source, so no way to spread this "in hardware". Windows didn't show such single-core load during test, so it seems it uses some kind of interrupt steering (spreads its load on the cores). And overall CPU load was perceived as lower in Windows host test than that in Linux host. This could not be directly compared, however.

The question is: why it sucks so much, am I missing something? Is it possible to have a performance comparable to that of Windows? (I am writing this with shaking hands and lost for words, it is very unpleasant to be catching-up in comparison with Windows.)

Additiontal tests as @shodanshok suggested:

[global]
ioengine=libaio
group_reporting
filename=/dev/vh0/testvol
direct=1
size=5G

[128K-Q1T32-Seq-Read]
rw=read
bs=128K
numjobs=32
stonewall

[128K-Q1T32-Seq-Write]
rw=write
bs=128K
numjobs=32
stonewall

[4K-Q1T32-Seq-Read]
rw=read
bs=4K
numjobs=32
stonewall

[4K-Q1T32-Seq-Write]
rw=write
bs=4K
numjobs=32
stonewall

[128K-Q1T2-Seq-Read]
rw=read
bs=128K
numjobs=2
stonewall

[128K-Q1T2-Seq-Write]
rw=write
bs=128K
numjobs=2
stonewall

The result:

128K-Q1T32-Seq-Read  bw=924MiB/s (969MB/s), 924MiB/s-924MiB/s (969MB/s-969MB/s), io=160GiB (172GB), run=177328-177328msec
128K-Q1T32-Seq-Write bw=441MiB/s (462MB/s), 441MiB/s-441MiB/s (462MB/s-462MB/s), io=160GiB (172GB), run=371784-371784msec
4K-Q1T32-Seq-Read    bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=160GiB (172GB), run=627761-627761msec
4K-Q1T32-Seq-Write   bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=160GiB (172GB), run=1240437-1240437msec
128K-Q1T2-Seq-Read   bw=427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=10.0GiB (10.7GB), run=23969-23969msec
128K-Q1T2-Seq-Write  bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=10.0GiB (10.7GB), run=22498-22498msec

Things are strange, why 128K-Q1T2-Seq-Read was so bad? (The ideal value is 1200MB/s.) 5 GiB per job is too small to have things settled? Everything else seems to be ok.

This looks very strange. I use Bacula and now BareOS for more than 10 years, but now one system experiences strange behaviour and I can't find out why and how to fix.

When it runs daily backups, it works fine, until it reaches the BackupCatalog job, which is configured to run after everything else.

This job looks like it was successfully terminated (JobStatus=T in the list jobs table):

*list jobs
...
+-------+---------------+--------------+---------------------+------+-------+----------+-----------------+-----------+
| JobId | Name          | Client       | StartTime           | Type | Level | JobFiles | JobBytes        | JobStatus |
+-------+---------------+--------------+---------------------+------+-------+----------+-----------------+-----------+
...
| 5,475 | BackupCatalog | kantor-fd    | 2019-12-04 02:56:40 | B    | F     |       21 |      27,364,860 | T         |
+-------+---------------+--------------+---------------------+------+-------+----------+-----------------+-----------+

However, in messages and in the log file I don't see the usual summary for this last job. The log file finishes like this:

19-Nov 02:32 kantor-dir JobId 5398: shell command: run BeforeJob "/usr/lib/bareos/scripts/make_catalog_backup.pl Kantor"
19-Nov 02:33 kantor-dir JobId 5398: Start Backup JobId 5398, Job=BackupCatalog.2019-11-18_23.10.00_10
19-Nov 02:33 kantor-dir JobId 5398: Using Device "FileStorage" to write.
19-Nov 02:33 kantor-sd JobId 5398: Volume "Kantor-2018-01-08_08:48:50" previously written, moving to end of data.
19-Nov 02:33 kantor-sd JobId 5398: Ready to append to end of Volume "Kantor-2018-01-08_08:48:50" size=4716094462
19-Nov 02:33 kantor-sd JobId 5398: Elapsed time=00:00:05, Transfer rate=5.663 M Bytes/second

And that's all. Note, RunAfterJob script seem to had not been executed. But if I execute it by hand, it works (the exported catalog database file gets removed). This is not the only job with RunAfterJob script, however.

I expected it to show something like this in the end. All other jobs have it:

19-Nov 02:32 kantor-dir JobId 5397: Bareos kantor-dir 16.2.6 (02Jun17):
  Build OS:               x86_64-pc-linux-gnu debian Debian GNU/Linux buster/sid
  JobId:                  5397
  Job:                    FTP.2019-11-18_23.05.00_09
...
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK

19-Nov 02:32 kantor-dir JobId 5397: Begin pruning Jobs older than 1 month 10 days .
...

Also, the director status looks strange:

*status dir
kantor-dir Version: 16.2.6 (02 June 2017) x86_64-pc-linux-gnu debian Debian GNU/Linux buster/sid
Daemon started 03-Dec-19 11:10. Jobs: run=4, running=1 mode=0 db=mysql
 Heap: heap=135,168 smbytes=222,459 max_bytes=236,758 bufs=543 max_bufs=594

Scheduled Jobs:
...
====

Running Jobs:
Console connected at 04-Dec-19 09:03
 JobId Level   Name                       Status
======================================================================
  5475 Full    BackupCatalog.2019-12-03_23.10.00_08 has terminated
====

Terminated Jobs:

 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
...
  5471  Incr      6,591    7.499 G  OK       03-Dec-19 23:15 termsrv
  5472  Incr        427    11.37 G  OK       03-Dec-19 23:44 1C
  5473  Incr          3    3.198 G  OK       04-Dec-19 02:56 Oracle
  5474  Incr      5,797    2.600 G  OK       04-Dec-19 02:56 FTP


Client Initiated Connections (waiting for jobs):
...
====

i.e. the said job listed in the "running jobs", but it says it is terminated. It doesn't listed in the "terminated jobs", as if director still has something to finish.

It hanged in this state for six hours. I also see some strangeness with times (StartTime for it in the table and in the log file differs by half of hour, however, system date and MySQL's select NOW(); are in sync).

After director restart, director status looks more appropriate:

Running Jobs:
Console connected at 04-Dec-19 09:06
No Jobs running.
====
No Terminated Jobs.

This all started two weeks ago. If I leave it hanging, all following scheduled jobs will wait indefinitely for this stuck job, meaning, no backups will be performed.

I feel this could be issue with this job's RunAfterJob script, but it is standard shipped script. If I run in by the hand, it works. The job definition itself is also standard shipped, the only modification is that I added compression=GZIP in the FileSet, but I do that every time and this never caused any problems.

What to look for? How to fix?

Update:

The problem disappeared. I don't understand, why. Backups work for at least two days. Nothing appears to be stuck.

I've installed a recent Debian 10 Buster system on to the server, which is HPE DL360 Gen9. It has P440ar adapter, which works with the "new" hpsa driver. As far as I remember, RAIDs were configured with built-in "pre-boot" GUI utility. All firmwares were updated to their most recent versions, so I believe that utility was also most recent version.

Now I have to set up RAID state monitoring to a Zabbix server.

hpsa arrays are managed with the hpssacli utility (the old hpacucli supports a cciss driver, which isn't applicable for me). I have a wrapper script which runs from Zabbix agent, it is able to discover and query state of each array in the system, that script just calls hpssacli, parses and adapts its output for Zabbix. I've doing that for ages.

On this newly set up system, I have a trouble. I tried a HPe's own SDR MCP repo, it doesn't support buster yes (HPe is notoriusly slow in updating their repos), so I just found a most recent hpssacli deb and installed it. It appeared to be hpssacli-2.40-13.0_amd64.deb, dated 2016-06-28 17:55.

However, when I tried to run it, it says: my array was created with more recent version of utility and my version is too old to manage it:

root@vh3:~# wget https://downloads.linux.hpe.com/SDR/repo/mcp/pool/non-free/hpssacli-2.40-13.0_amd64.deb
--2019-11-25 14:13:38--  https://downloads.linux.hpe.com/SDR/repo/mcp/pool/non-free/hpssacli-2.40-13.0_amd64.deb
Распознаётся downloads.linux.hpe.com (downloads.linux.hpe.com)… 15.249.152.85
Подключение к downloads.linux.hpe.com (downloads.linux.hpe.com)|15.249.152.85|:443... соединение установлено.
HTTP-запрос отправлен. Ожидание ответа… 200 OK
Длина: 8237034 (7,9M)
Сохранение в: «hpssacli-2.40-13.0_amd64.deb»

hpssacli-2.40-13.0_amd64.deb                    100%[====================================================================================================>]   7,85M   394KB/s    за 22s     

2019-11-25 14:14:01 (363 KB/s) - «hpssacli-2.40-13.0_amd64.deb» сохранён [8237034/8237034]

root@vh3:~# ls
hpssacli-2.40-13.0_amd64.deb
root@vh3:~# dpkg -i hpssacli-2.40-13.0_amd64.deb 
Выбор ранее не выбранного пакета hpssacli.
(Чтение базы данных … на данный момент установлено 57199 файлов и каталогов.)
Подготовка к распаковке hpssacli-2.40-13.0_amd64.deb …
Распаковывается hpssacli (2.40-13.0) …
Настраивается пакет hpssacli (2.40-13.0) …
Обрабатываются триггеры для man-db (2.8.5-2) …
root@vh3:~# hpssacli ctrl all show

Smart Array P440ar in Slot 0 (Embedded) 

APPLICATION UPGRADE REQUIRED: This controller has been configured with a more
                              recent version of software.
                              To prevent data loss, configuration changes to
                              this controller are not allowed.
                              Please upgrade to the latest version to be able
                              to continue to configure this controller.

While this doesn't seem to prevent my script to monitor a controller state, I also want to be able to manage it from the operating system, to be able to add drives and create more arrays without stopping a system in the future.

I also tried to use hwraid.le-vert.net repo, but there is no hpssacli utility (it only has hpacucli, at least in buster).

What I supposed to do? Where to find this "latest" version and how to find out which version I need?

we are setting up a centralized logging server which receives logs from many devices and servers and stored them into a database.

The syslog on the central server is syslog-ng. It is chosen because of simplicity of its configuration and easier manageability. The log server itself hosts log database (mariadb) and web interface (apache/mod_php).

The whole point of having a log server is to be able to analyze data in a single place. For that we chose a Adiscon LogAnalyzer, which looks like maintained, mature and feature-rich project.

LogAnalyzer manual recommends to use a MonitorWare schema. For example, installation manual has a "Note on MySQL Databases" section that says it supports "php-syslog-ng" schema for transition purposes and won't utilize it fully, and that schema is not very efficient, and recommends to use MonitorWare schema instead. (MonitorWare is another product by Adiscon.)

Also, it recommends to use rsyslog as it should create structure in the database automatically (rsyslog is also seems to be supported by Adiscon, what an incidence). Aside from that, nowhere this schema is particularly documented.

There is a question from 2011 on how to configure syslog-ng with this schema. This is also one of few places where schema is presented in clear, but it links to another resource which is currently unavailable (and archive.org knows nothing about it either).

So the question remains unanswered: how to properly set up Syslog-NG logging into database with MonitorWare schema?

I need to expand disk capacity of the server. The pool was started with 1Tb disk, then it was expanded with 2Tb disk. There is more than 1Tb free space, i.e. all data will easily fit 2Tb part, but it is currently allocated on 1Tb disks.

In reality, these disks are hardware (PERC) RAID1 arrays over pair of 1Tb and pair of 2Tb resp.

I want to replace these 1Tb disks with 3Tb. The "one by one" replacement isn't really an option. In principle, this physical RAID can replace disks one by one, then grow the array to fill disks. However, I want to avoid this path, because it leaves some quite long time periods when disk redundancy is lost.

I want to literally move all data off from 1Tb, then remove it, and replace with 3Tb. All to be done on the fly, with running system, with zero downtime.

With LVM the operation has to be very straightforward and easy to understand:

1. pvmove all allocated data from 1Tb physical disk (VD in RAID terms)
2. vgreduce to remove that pv from vg and pvremove to remove pv metadata
3. remove a 1Tb array using megacli (PERC is rebranded LSI/Avago MegaRAID SAS)
4. physically replace disks
5. assemble another array again using megacli
6. make a new pv and add it into vg

This is the routine procedure I used to do. Each step is very well understood, on each step I have full control of what is going on, I always know how to proceed if something goes wrong and so on.

How to do the same procedure safely and knowingly with ZFS?

If that matters:

• the server is Dell PowerEdge R730
• the OS is Proxmox VE 6.0, which is based on Debian 10.1. It was installed from PVE ISO image, i.e. not converted from Debian installation.
• the system doesn't depend on this this pool, because it runs from set of SSDs assembled into another pool
• the pool hosts some VM virtual disks which don't require high performance. However, that data is valuable, I can't tolerate the if it's lost. Therefore the procedure should be clear and understandable
• system is constantly used by users, but they will tolerate the performance loss during data migration

Checkpoint firewall have two NAT modes: Static NAT (1-to-1 internal IP to external IP) and Hide NAT, which is called "overload" in the Cisco and so on.

It is not being said in clear, but it seems that Static NAT doesn't change any port numbers, only IPs.

The Hide NAT, in contrast, changes source port on outgoing packet to some random "dynamically assigned", and then installs a rule in the table which allows for reverse packet translation. This rule is removed on clear connection end (TCP FIN, RST) or timeout. This is what basically all current NAT-capable devices always do.

I have only one external IP address, so I am required to use Hide NAT variant. However, I want to augment it slightly, as I have a very special host inside, which runs a very special service, which doesn't particularly like to have their source port number changed and randomly selected. I need to have a standard Hide NAT, which is using for "dynamic allocation" all ports except one (say don't translate anything to UDP 33333), and packets of this particular service from this particular port should be always and only translated to this port number.

In Linux I easily do this with these rules:

iptables -t nat -s MY-SPECIAL-HOST -p udp --sport MY-SPECIAL-SERVICE-PORT -j SNAT --to-source MY-WHITE-IP:MY-SPECIAL-SERVICE-PORT
iptables -t nat -s MY-LOCAL-NETWORK -p udp -j SNAT --to-source MY-WHITE-IP:10000-[MY-SPECIAL-SERVICE-PORT - 1]
iptables -t nat -s MY-LOCAL-NETWORK -j SNAT --to-source MY-WHITE-IP

(I know I limit this number of ports for outgoing UDP conversations, this is not an issue, as the actual service port is somewhere in the end of the range.)

How to achieve the same in the Checkpoint? I have GAIA R77.10 ClusterXL with two members, if this is needed.

I've had run a Galera Cluster for a while, MariaDB version 10.0.20, Galera version 25.3.10, everything over Gentoo. That cluster worked fine, nodes restarted several times for maintenance and so on. When I started mysql on any node, it loaded wsrep_provider library, which started SST and then eventually joined the cluster.

Now the time come to do massive upgrade. I have chosen a "rolling upgrade" path described in http://galeracluster.com/documentation-webpages/upgrading.html#id1 , take the chosen node down and update software there.

For impatient, a short summary of what follows: mariadb starts and ignores any wsrep_ config options, and if I specify them by hand in running instance, it seems to hang.

Two questions: - Why it silently ignores my configuration options? How to make it respect them? - What to do to start replication, how to join a node into cluster?

Everything upraded fine. Now I have MariaDB 10.1.13, Galera 25.3.15. Configuration in /etc/mysql/my.cnf was merged. It now looks like this (only mysqld section):

[mysqld]
character-set-server            = utf8
user                            = mysql
port                            = 3306
socket                          = /var/run/mysqld/mysqld.sock
pid-file                        = /var/run/mysqld/mysqld.pid
log-error                       = /var/log/mysql/mysqld.err
basedir                         = /usr
datadir                         = /var/lib/mysql
skip-external-locking
key_buffer_size                 = 16M
max_allowed_packet              = 4M
table_open_cache                = 400
sort_buffer_size                = 512K
net_buffer_length               = 16K
read_buffer_size                = 256K
read_rnd_buffer_size            = 512K
myisam_sort_buffer_size         = 8M
lc_messages_dir                 = /usr/share/mysql
lc_messages                     = en_US

log-bin
server-id                       = 1

tmpdir                          = /tmp/

innodb_buffer_pool_size = 128M
innodb_data_file_path = ibdata1:10M:autoextend:max:128M
innodb_log_file_size = 48M
innodb_log_buffer_size = 8M
innodb_log_files_in_group=2
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_file_per_table

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="www_cluster"
wsrep_cluster_address="gcomm://192.168.4.28,192.168.5.28,192.168.5.29"
wsrep_sst_method=rsync

I triple-checked this file, to be sure my wsrep_* options are in [mysqld] section as they were. These options are exactly like they were before upgrade.

However, mysql service is started now as a normal standalone server, not clustered:

2016-04-04 16:30:46 139950286550848 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used;  This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=www1' or '--log-bin=mysqld-bin' to avoid this problem.
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: The InnoDB memory heap is disabled
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Memory barrier is not used
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Compressed tables use zlib 1.2.8
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using Linux native AIO
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using generic crc32 instructions
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Completed initialization of buffer pool
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Highest supported file format is Barracuda.
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: 128 rollback segment(s) are active.
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: Waiting for purge to start
2016-04-04 16:30:47 139950286550848 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.28-76.1 started; log sequence number 116616176
2016-04-04 16:30:47 139949849761536 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-04-04 16:30:47 139950286550848 [Note] Plugin 'FEEDBACK' is disabled.
2016-04-04 16:30:47 139950286550848 [Note] Server socket created on IP: '0.0.0.0'.
2016-04-04 16:30:47 139950286550848 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.13-MariaDB'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Source distribution

If I query wsrep_* variables it shows no provider loaded:

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+--------------------------+----------------------+
| Variable_name            | Value                |
+--------------------------+----------------------+
| wsrep_cluster_conf_id    | 18446744073709551615 |
| wsrep_cluster_size       | 0                    |
| wsrep_cluster_state_uuid |                      |
| wsrep_cluster_status     | Disconnected         |
| wsrep_connected          | OFF                  |
| wsrep_local_bf_aborts    | 0                    |
| wsrep_local_index        | 18446744073709551615 |
| wsrep_provider_name      |                      |
| wsrep_provider_vendor    |                      |
| wsrep_provider_version   |                      |
| wsrep_ready              | OFF                  |
| wsrep_thread_count       | 0                    |
+--------------------------+----------------------+
12 rows in set (0.00 sec)

It looks like it didn't respect a wsrep_provider and wsrep_cluster_address (and I believe all other wsrep_*) options.

I tried to set them globally in running mariadb instance by hand:

MariaDB [(none)]> set global wsrep_provider='/usr/lib/galera/libgalera_smm.so';
Query OK, 0 rows affected (0.02 sec)
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+-----------------------------------+
| Variable_name                | Value                             |
+------------------------------+-----------------------------------+
| wsrep_apply_oooe             | 0.000000                          |
| wsrep_apply_oool             | 0.000000                          |
| wsrep_apply_window           | 0.000000                          |
| wsrep_causal_reads           | 0                                 |
| wsrep_cert_deps_distance     | 0.000000                          |
| wsrep_cert_index_size        | 0                                 |
| wsrep_cert_interval          | 0.000000                          |
| wsrep_cluster_conf_id        | 18446744073709551615              |
| wsrep_cluster_size           | 0                                 |
| wsrep_cluster_state_uuid     |                                   |
| wsrep_cluster_status         | Disconnected                      |
| wsrep_commit_oooe            | 0.000000                          |
| wsrep_commit_oool            | 0.000000                          |
| wsrep_commit_window          | 0.000000                          |
| wsrep_connected              | OFF                               |
| wsrep_flow_control_paused    | 0.000000                          |
| wsrep_flow_control_paused_ns | 0                                 |
| wsrep_flow_control_recv      | 0                                 |
| wsrep_flow_control_sent      | 0                                 |
| wsrep_incoming_addresses     |                                   |
| wsrep_last_committed         | 18446744073709551615              |
| wsrep_local_bf_aborts        | 0                                 |
| wsrep_local_cached_downto    | 18446744073709551615              |
| wsrep_local_cert_failures    | 0                                 |
| wsrep_local_commits          | 0                                 |
| wsrep_local_index            | 18446744073709551615              |
| wsrep_local_recv_queue       | 0                                 |
| wsrep_local_recv_queue_avg   | 0.000000                          |
| wsrep_local_recv_queue_max   | 0                                 |
| wsrep_local_recv_queue_min   | 0                                 |
| wsrep_local_replays          | 0                                 |
| wsrep_local_send_queue       | 0                                 |
| wsrep_local_send_queue_avg   | 0.000000                          |
| wsrep_local_send_queue_max   | 0                                 |
| wsrep_local_send_queue_min   | 0                                 |
| wsrep_local_state            | 0                                 |
| wsrep_local_state_comment    | Initialized                       |
| wsrep_local_state_uuid       |                                   |
| wsrep_protocol_version       | 18446744073709551615              |
| wsrep_provider_name          | Galera                            |
| wsrep_provider_vendor        | Codership Oy <[email protected]> |
| wsrep_provider_version       | 3.15(r8459459)                    |
| wsrep_ready                  | OFF                               |
| wsrep_received               | 0                                 |
| wsrep_received_bytes         | 0                                 |
| wsrep_repl_data_bytes        | 0                                 |
| wsrep_repl_keys              | 0                                 |
| wsrep_repl_keys_bytes        | 0                                 |
| wsrep_repl_other_bytes       | 0                                 |
| wsrep_replicated             | 0                                 |
| wsrep_replicated_bytes       | 0                                 |
| wsrep_thread_count           | 0                                 |
+------------------------------+-----------------------------------+
52 rows in set (0.00 sec)

respective log entries:

2016-04-04 16:45:19 139950170823424 [Note] WSREP: Stop replication
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Provider was not loaded, in stop replication
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Initial position: 913d310a-9360-11e4-9d31-922a1e5d98fe:63026
2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): Galera 3.15(r8459459) by Codership Oy <[email protected]> loaded successfully.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.4.28; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false;
2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1

and fill up cluster addresses:

MariaDB [(none)]> set global wsrep_cluster_address='gcomm://192.168.4.28,192.168.5.28,192.168.5.29';    
Query OK, 0 rows affected (3.55 sec)

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+-------------------------------------------------------+
| Variable_name                | Value                                                 |
+------------------------------+-------------------------------------------------------+
| wsrep_apply_oooe             | 0.000000                                              |
| wsrep_apply_oool             | 0.000000                                              |
| wsrep_apply_window           | 0.000000                                              |
| wsrep_causal_reads           | 0                                                     |
| wsrep_cert_deps_distance     | 0.000000                                              |
| wsrep_cert_index_size        | 0                                                     |
| wsrep_cert_interval          | 0.000000                                              |
| wsrep_cluster_conf_id        | 471                                                   |
| wsrep_cluster_size           | 3                                                     |
| wsrep_cluster_state_uuid     | 913d310a-9360-11e4-9d31-922a1e5d98fe                  |
| wsrep_cluster_status         | Primary                                               |
| wsrep_commit_oooe            | 0.000000                                              |
| wsrep_commit_oool            | 0.000000                                              |
| wsrep_commit_window          | 0.000000                                              |
| wsrep_connected              | ON                                                    |
| wsrep_evs_delayed            |                                                       |
| wsrep_evs_evict_list         |                                                       |
| wsrep_evs_repl_latency       | 0.0456547/0.049131/0.0560754/0.00491047/3             |
| wsrep_evs_state              | OPERATIONAL                                           |
| wsrep_flow_control_paused    | 0.000000                                              |
| wsrep_flow_control_paused_ns | 0                                                     |
| wsrep_flow_control_recv      | 0                                                     |
| wsrep_flow_control_sent      | 0                                                     |
| wsrep_gcomm_uuid             | fd416235-fa6b-11e5-867c-72011d51ce8d                  |
| wsrep_incoming_addresses     | 192.168.5.28:3306,192.168.5.29:3306,192.168.4.28:3306 |
| wsrep_last_committed         | 18446744073709551615                                  |
| wsrep_local_bf_aborts        | 0                                                     |
| wsrep_local_cached_downto    | 18446744073709551615                                  |
| wsrep_local_cert_failures    | 0                                                     |
| wsrep_local_commits          | 0                                                     |
| wsrep_local_index            | 2                                                     |
| wsrep_local_recv_queue       | 0                                                     |
| wsrep_local_recv_queue_avg   | 0.000000                                              |
| wsrep_local_recv_queue_max   | 1                                                     |
| wsrep_local_recv_queue_min   | 0                                                     |
| wsrep_local_replays          | 0                                                     |
| wsrep_local_send_queue       | 0                                                     |
| wsrep_local_send_queue_avg   | 0.000000                                              |
| wsrep_local_send_queue_max   | 1                                                     |
| wsrep_local_send_queue_min   | 0                                                     |
| wsrep_local_state            | 1                                                     |
| wsrep_local_state_comment    | Joining: receiving State Transfer                     |
| wsrep_local_state_uuid       |                                                       |
| wsrep_protocol_version       | 7                                                     |
| wsrep_provider_name          | Galera                                                |
| wsrep_provider_vendor        | Codership Oy <[email protected]>                     |
| wsrep_provider_version       | 3.15(r8459459)                                        |
| wsrep_ready                  | OFF                                                   |
| wsrep_received               | 1                                                     |
| wsrep_received_bytes         | 266                                                   |
| wsrep_repl_data_bytes        | 0                                                     |
| wsrep_repl_keys              | 0                                                     |
| wsrep_repl_keys_bytes        | 0                                                     |
| wsrep_repl_other_bytes       | 0                                                     |
| wsrep_replicated             | 0                                                     |
| wsrep_replicated_bytes       | 0                                                     |
| wsrep_thread_count           | 2                                                     |
+------------------------------+-------------------------------------------------------+
57 rows in set (0.00 sec)

respective log:

2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-04-04 16:49:00 139950170520320 [Note] WSREP: Stop replication
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Start replication
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:49:02 139950170520320 [Note] WSREP: protonet asio version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Using CRC-32C for message checksums.
2016-04-04 16:49:02 139950170520320 [Note] WSREP: backend: asio
2016-04-04 16:49:02 139950170520320 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2016-04-04 16:49:02 139950170520320 [Note] WSREP: restore pc from disk failed
2016-04-04 16:49:02 139950170520320 [Note] WSREP: GMCast version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2016-04-04 16:49:02 139950170520320 [Note] WSREP: EVS version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: gcomm: connecting to group 'www_cluster', peer '192.168.4.28:,192.168.5.28:,192.168.5.29:'
2016-04-04 16:49:02 139950170520320 [Warning] WSREP: (fd416235, 'tcp://0.0.0.0:4567') address 'tcp://192.168.4.28:4567' points to own listening address, blacklisting
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring 4c4f4779 at tcp://192.168.5.28:4567 stable
2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Node 4c4f4779 state prim
2016-04-04 16:49:02 139950170520320 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb {
        4c4f4779,0
        b75760e5,0
        fd416235,0
} joined {
} left {
} partitioned {
})
2016-04-04 16:49:02 139950170520320 [Note] WSREP: save pc into disk
2016-04-04 16:49:03 139950170520320 [Note] WSREP: gcomm: connected
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Opened channel 'www_cluster'
2016-04-04 16:49:03 139948892088064 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2)
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera)
2016-04-04 16:49:03 139949710771968 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.55477S), skipping check
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 ()
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Quorum results:
        version    = 3,
        component  = PRIMARY,
        conf_id    = 470,
        members    = 2/3 (joined/total),
        act_id     = 63180,
        last_appl. = -1,
        protocols  = 0/7/3 (gcs/repl/appl),
        group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Flow-control interval: [28, 28]
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 63180)
2016-04-04 16:49:03 139950170217216 [Note] WSREP: State transfer required:
        Group state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180
        Local state: 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:49:03 139950170217216 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 2, protocol version 3
2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-04-04 16:49:03 139948883695360 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.4.28' --datadir '/var/lib/mysql/'  --defaults-file '/etc/mysql/my.cnf'  --parent '6067' --binlog 'mysqld-bin' '
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Prepared SST request: rsync|192.168.4.28:4444/rsync_sst
2016-04-04 16:49:03 139950170217216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-04-04 16:49:03 139950170217216 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3
2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (913d310a-9360-11e4-9d31-922a1e5d98fe): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 63180)
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Requesting state transfer: success, donor: 0
2016-04-04 16:49:05 139949710771968 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting off

And now node keep sitting in "Joining: receiving State Transfer".

Selected donor node (www2), however, says it had completed state transfer:

160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable
160404 16:49:04 [Note] WSREP: declaring fd416235 at tcp://192.168.4.28:4567 stable
160404 16:49:04 [Note] WSREP: Node 4c4f4779 state prim
160404 16:49:04 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb {
    4c4f4779,0
    b75760e5,0
    fd416235,0
} joined {
} left {
} partitioned {
})
160404 16:49:04 [Note] WSREP: save pc into disk
160404 16:49:04 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
160404 16:49:04 [Note] WSREP: STATE_EXCHANGE: sent state UUID: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2)
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera)
160404 16:49:05 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 ()
160404 16:49:05 [Note] WSREP: Quorum results:
    version    = 3,
    component  = PRIMARY,
    conf_id    = 470,
    members    = 2/3 (joined/total),
    act_id     = 63180,
    last_appl. = 63152,
    protocols  = 0/7/3 (gcs/repl/appl),
    group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe
160404 16:49:05 [Note] WSREP: Flow-control interval: [28, 28]
160404 16:49:05 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 0, protocol version 3
160404 16:49:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160404 16:49:05 [Note] WSREP: REPL Protocols: 7 (3, 2)
160404 16:49:05 [Note] WSREP: Service thread queue flushed.
160404 16:49:05 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3
160404 16:49:05 [Note] WSREP: Service thread queue flushed.
160404 16:49:05 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor.
160404 16:49:05 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 63180)
160404 16:49:06 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160404 16:49:06 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.4.28:4444/rsync_sst' --auth '(null)' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix ''  --binlog 'mysqld-bin' --gtid '913d310a-9360-11e4-9d31-922a1e5d98fe:63180''
160404 16:49:06 [Note] WSREP: sst_donor_thread signaled with 0
160404 16:49:06 [Note] WSREP: Flushing tables for SST...
160404 16:49:06 [Note] WSREP: Provider paused at 913d310a-9360-11e4-9d31-922a1e5d98fe:63180 (9555)
160404 16:49:06 [Note] WSREP: Tables flushed.
WSREP_SST: [INFO] Preparing binlog files for transfer: (20160404 16:49:07.005)
mysqld-bin.000033
160404 16:49:07 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting off
160404 16:51:39 [Note] WSREP: resuming provider at 9555
160404 16:51:39 [Note] WSREP: Provider resumed.
160404 16:51:40 [Note] WSREP: 0.0 (www2): State transfer to 2.0 () complete.
160404 16:51:40 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 63180)
160404 16:51:40 [Note] WSREP: Member 0.0 (www2) synced with group.
160404 16:51:40 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 63180)
160404 16:51:40 [Note] WSREP: Synchronized with group, ready for connections
160404 16:51:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

What to do now? How do I continue update node operation?