SvennD's questions

I would like to forward snmp from one LAN to another LAN. I believe I got one way working (from outside to inside), but sadly the box1 cannot seem to reply. (inside to outside)

The setup I would like to make :

  box1                  box2             box3
[10.1.255.245] -------------------------- [public_ip]
    |               eth0-|-eth1               |
    |-----------3161---------------- 3161 ----|

These are the iptables rules which are relevant of BOX 2,

-A PREROUTING -i eth1 -p udp -m udp --dport 3161 -j DNAT --to-destination 10.1.255.244:3161
-A FORWARD -d 10.1.255.244/32 -p udp -m udp --dport 3161 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT

Now I validated that packages coming from box3 are received in box1.

tcpdump 'port 3161'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:46:03.887718 IP x-20150810-092601.x.be.53329 > compute-0-2.local.doc1lm: UDP, length 48
13:46:04.888851 IP x-20150810-092601.x.be.53329 > compute-0-2.local.doc1lm: UDP, length 48
13:46:05.889977 IP x-20150810-092601.x.be.53329 > compute-0-2.local.doc1lm: UDP, length 48
13:46:06.891108 IP x-20150810-092601.x.be.53329 > compute-0-2.local.doc1lm: UDP, length 48

So I assume the snmp process cannot respond to the location cause it does not know how to find this out-of-lan ip... how can I add this ? this is route on box1 :

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
255.255.255.255 *               255.255.255.255 UH    0      0        0 eth0
box2 box2     255.255.255.255 UGH   0      0        0 eth0
224.0.0.0       *               255.255.255.0   U     0      0        0 eth0
10.1.0.0        *               255.255.0.0     U     0      0        0 eth0
link-local      *               255.255.0.0     U     1002   0        0 eth0
link-local      *               255.255.0.0     U     1003   0        0 eth1
192.168.0.0     *               255.255.0.0     U     0      0        0 eth1
default         storage          0.0.0.0         UG    0      0        0 eth1

Or am I missing something else obvious ?

I tried to setup iptables to forward a port (161) of a machine (box1) in subnet 10.10.255.* to port 162 to box2, this should be queried by box3 that is in a different subnet (public one). This is a schematic :

  box1                  box2             box3
[10.1.255.245] ---- (eth0)-(eth1) ---- [public_ip]
    |                     |--------- 161 -----|
    |-----------161---(iptables)---- 162 -----|

So basically I want to query snmp on box3 through port 162. On box2 I added these rules:

iptables -t nat -A PREROUTING -p udp -i eth1 --dport 162 -j DNAT --to-destination 10.1.255.245:161
iptables -A FORWARD -p udp -d 10.1.255.245 --dport 161 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

Sadly box3 can't query snmp ... these are the iptables from box2 :

# Generated by iptables-save v1.4.7 on Tue Mar 14 10:41:21 2017
*filter
:INPUT ACCEPT [60:9648]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [421336:836276554]
-A INPUT -p udp -m udp --dport 161 -j ACCEPT
-A INPUT -p tcp -m tcp --sport 1024:65535 --dport 372 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p udp -m udp --sport 1024:65535 --dport 372 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p udp -m udp --dport 8649 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -m tcp --dport 3306 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -m tcp --dport 40000 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i eth1 -p tcp -m tcp --dport 0:1023 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i eth1 -p udp -m udp --dport 0:1023 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i eth1 -o eth0 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-A FORWARD -d 10.1.255.245/32 -p udp -m udp --dport 161 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Tue Mar 14 10:41:21 2017
# Generated by iptables-save v1.4.7 on Tue Mar 14 10:41:21 2017
*nat
:PREROUTING ACCEPT [397:45293]
:POSTROUTING ACCEPT [8465:1184160]
:OUTPUT ACCEPT [8465:1184160]
-A PREROUTING -i eth1 -p udp -m udp --dport 162 -j DNAT --to-destination 10.1.255.245:161
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
# Completed on Tue Mar 14 10:41:21 2017

Route on box 1 :

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
255.255.255.255 *               255.255.255.255 UH    0      0        0 eth0
box2            box2.local      255.255.255.255 UGH   0      0        0 eth0
224.0.0.0       *               255.255.255.0   U     0      0        0 eth0
10.1.0.0        *               255.255.0.0     U     0      0        0 eth0
link-local      *               255.255.0.0     U     1002   0        0 eth0
link-local      *               255.255.0.0     U     1003   0        0 eth1
192.168.0.0     *               255.255.0.0     U     0      0        0 eth1
default         storage1        0.0.0.0         UG    0      0        0 eth1

tcpdump of port 161 on box 2 :

2017-03-14 11:25:00.677570 IP box3.57060 > box2.162:  C=servers GetNextRequest(28)  .1.3.6.1.2.1.1.1.0

tcpdump of port 162 on box 2 :

2017-03-14 11:23:03.325818 IP box3.42352 > box1.161:  C=servers GetNextRequest(28)  .1.3.6.1.2.1.1.1.0

• I tried snmpwalk from box2 to box1 over 161 and that works
• /proc/sys/net/ipv4/ip_forward = 1 on box 2

What am I missing ?

We will have a machine at work, that on peak performance, should be able to push 50 ("write heads") x 75GB of data per hour. That's peak performance of ~1100MB/s write speed. To get that from the machine, it requires two 10GBi lines. My question is what kind of server+technology can handle/store such data flow ?

Currently for data storage we work with ZFS, although write speeds were never a question. (we are not even close to these speeds) Would ZFS (zfs on linux) be an option ? We also need to store a lot of data, the "IT guide" suggests somewhere between 50-75 TB in total. So it probably can't be all SSDs unless we want to offer our first-born child.

Some additions based on the excellent replies :

• the maximum is 50x75GB/hour during peak which is less than 24h (most likely <6h)
• We don't expect this to happen soon, most likely we will run 5-10x75GB/hour
• it's a pre-alpha machine, however requirements should be met (even though a lot of question marks are in play)
• we would use NFS as connection from the machine to the server
• layout : generating machine -> storage (this one) -> (safe raid 6) -> compute cluster
• so read speed is not essential, but it would be nice to use it from the compute cluster (but this is completely optional)
• most likely it's going to be large data files (not many small)

We have two large storage servers (+100TB), one runs on ZFS the other one runs XFS, we intent to use the XFS as our work server and use the ZFS as the backup server (snapshots <3). Now the problem is keeping these beasts in sync ... (sync as in daily synced)

The easiest option is to use rsync, but sadly the directory structure is deep, and full of hard links, all over the place. So this means we would need to do a "global" scan which would take ages... On top of that, most of the data is created and never modified. So rsync might just not be the way to go.

I looked into inotify, which seems relatively cheap and since we only wanne sync on a daily base, would be able to offload to a good time... sadly, if we only look to the created files we would copy hard links as data which would double the amount of storage used in our backup ... (basically there is no way to do the -H check from rsync)

The only option left I could think off would be to reorganize our storage to use a date based directory, sadly moving around that much data is not something we would prefer ...

Are there other options ?

For reference :

• Server with XFS has a raid controller (no JBOD option), and SATA disks (WD RE). 32Gb RAM
• Server with ZFS has a HBA controller and SAS disks. 126Gb RAM

When I reference ZFS as beeing slow, I see 'ls' taking seconds...

So I have been searching for a few days to set up this "easy" thing. I want to have a file server where users can login with there domain name (domain\name) and password. The server should give them access to a single directory based on their name. A few admin's (or local users) should be able to access them. This worked, however I was working on a virtual machine, and following multiple half deprecated guides. So I replicated my steps on a fresh machine but can't get it to work. (both Centos 7.2 up-to-date)

My method in short :

• set up timezone
• install krb5-workstation and set it up kinit [email protected]
• install samba samba-winbind-clients set it up (see config below)
• net ads join -U [email protected] (successful; see below)
• start services smb, nmb and winbind
• added winbind in /etc/nsswitch.conf to passwd, shadow and group
• tested domain users : wbinfo -n user and wbinfo -g (groups)

the result of join :

Enter [email protected]'s password:
Using short domain name -- DOMAIN
Joined 'SERVER' to dns domain 'domain.url'
No DNS domain configured for server. Unable to perform DNS Update.
DNS update failed: NT_STATUS_INVALID_PARAMETER

config for samba :

[global]
        netbios name = SERVER_NAME
        workgroup = DOMAIN
        realm = DOMAIN.url
        server string = Samba Server Version %v
        security = ADS
        allow trusted domains = No
        obey pam restrictions = Yes
        password server = first.domain.controler.url
        log file = /var/log/samba/log.%m
        max log size = 50
        load printers = No
        preferred master = No
        idmap backend = idmap_rid:acme=16777216-33554431
        template homedir = /home/%U
        template shell = /bin/bash
        winbind separator = +
        winbind enum users = Yes
        winbind enum groups = Yes
        winbind use default domain = Yes
        idmap config * : range = 16777216-33554431
        idmap config * : backend = idmap_rid:acme=16777216-33554431
        cups options = raw

[microsphere]
        root preexec = bash -c '[[ -d /data/%U ]] || mkdir -m 0700 /data/%U && chown %U:"Domain Users" /data/%U'
        comment = Home Directories
        valid users = "@DOMAIN+Domain Users"
        admin users = "@DOMAIN+Domain Admins"
        path = /data
        read only = no
        create mask = 0600
        force create mode = 0600
        directory mask = 0700
        force directory mode = 0700
        hide unreadable = Yes
        access based share enum = Yes

All of this seems to work oke :

wbinfo -n test
S-1-5-21-999108875-1658920850-184960113-4061 SID_USER (1)

Check if group is there : getent group "Domain Users" domain users:x:4294967295:

However on the "clean" server, the permissions are never set correctly. The directory is created, but the permissions are not set correctly. Also doing them manually refuses to. (note : /data is on the root partition, nothing external)

[root@server data]# chown -v test."domain users" test/
ownership of ‘test/’ retained as root:root
[root@server data]# ll
total 1
drwx------ 2 root root 3 Aug 11 09:43 svenn
drwx------ 2 root root 2 Aug 11 09:48 test

Whats going on ?

I recently upgraded my server to Centos 7; I asked the users to backup there data for the upgrade. After the upgrade some user tried to untar a backup tar (~300Gb) and this caused the server to hang / freeze. (not responding to keyboard/ssh input) (reproducable) This happened as well with a other binary that was resource intensive.

I monitored memory and CPU and both are reasonable until the freeze, so the conclusion on the Centos Board was that it is I/0 limitation. This seemed correct as the workaround nice finished the untar succesfully.

My question is, how can I be certain that this does not happen again in the future ? (Can I limit the I/0 ?) This is a "offline" machine so users expect a large freedom to experiment. (but obviously we don't want to reset the machine every day.) Can I nice processes by default ? Or limit the I/0 so that the machine does not "freeze" ? The system is running of a raidcontroller that has raid 6 on 6 x Western Digital RE4

Specs for reference : (while workload was niced to 19)

iotop : (max about 60 M/s)

Total DISK READ :      51.72 M/s | Total DISK WRITE :      42.40 M/s
Actual DISK READ:     786.07 K/s | Actual DISK WRITE:     131.94 K/s

top while :

top - 09:58:27 up 1 day, 26 min,  6 users,  load average: 142.30, 144.85, 148.19
Tasks: 785 total,   8 running, 777 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  6.5 sy, 93.0 ni,  0.0 id,  0.1 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem : 52838147+total,   707112 free, 51992025+used,  7754104 buff/cache
KiB Swap:  4194300 total,        0 free,  4194300 used.  7746868 avail Mem

free -m

              total        used        free      shared  buff/cache   available
Mem:         515997      507679         719           3        7598        7631
Swap:          4095        4095           0

cat /proc/cpuinfo

processor       : 63
vendor_id       : GenuineIntel
cpu family      : 6
model           : 47
model name      : Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz
stepping        : 2
microcode       : 0x37
cpu MHz         : 2134.000
cache size      : 24576 KB
physical id     : 3
siblings        : 16
core id         : 25
cpu cores       : 8
apicid          : 243
initial apicid  : 243
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt aes lahf_lm ida arat epb dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips        : 4266.05
clflush size    : 64
cache_alignment : 64
address sizes   : 44 bits physical, 48 bits virtual
power management:

lspci (raid controller)

02:00.0 RAID bus controller: 3ware Inc 9750 SAS2/SATA-II RAID PCIe (rev 05)

system :

cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
uname -r
3.10.0-327.el7.x86_64

 lspci
00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
02:00.0 RAID bus controller: 3ware Inc 9750 SAS2/SATA-II RAID PCIe (rev 05)
04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
05:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
fc:00.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 1 (rev 0c)
fc:00.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 2 (rev 0c)
fc:00.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 3 (rev 0c)
fc:00.6 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 4 (rev 0c)
fc:01.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Power Controller (rev 0c)
fc:02.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 0 (rev 0c)
fc:03.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 1 (rev 0c)
fc:04.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 0 (rev 0c)
fc:05.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0a (rev 0c)
fc:05.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0b (rev 0c)
fc:05.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0c (rev 0c)
fc:05.6 Host bridge: Intel Corporation Device 2b53 (rev 0c)
fc:06.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 1 (rev 0c)
fc:07.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1a (rev 0c)
fc:07.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1b (rev 0c)
fc:07.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1c (rev 0c)
fc:07.6 Host bridge: Intel Corporation Device 2b5b (rev 0c)
fc:08.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 0 (rev 0c)
fc:09.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 1 (rev 0c)
fc:0a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 2 (rev 0c)
fc:0b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 3 (rev 0c)
fc:0c.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 4 (rev 0c)
fc:0d.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 5 (rev 0c)
fc:0e.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 6 (rev 0c)
fc:0f.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 7 (rev 0c)
fc:10.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 0-1 (rev 0c)
fc:10.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 2-3 (rev 0c)
fc:10.4 Host bridge: Intel Corporation Device 2b32 (rev 0c)
fc:10.6 Host bridge: Intel Corporation Device 2b3a (rev 0c)
fc:11.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 4-5 (rev 0c)
fc:11.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 6-7 (rev 0c)
fc:11.4 Host bridge: Intel Corporation Device 2b36 (rev 0c)
fc:11.6 Host bridge: Intel Corporation Device 2b3e (rev 0c)
fc:12.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 0 (rev 0c)
fc:13.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 1 (rev 0c)
fc:14.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: REUT control/status (rev 0c)
fc:14.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: Misc. control/status (rev 0c)
fc:15.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: REUT control/status (rev 0c)
fc:15.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: Misc. control/status (rev 0c)
fc:17.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: REUT control/status (rev 0c)
fc:17.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: Misc. control/status (rev 0c)
fc:18.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: REUT control/status (rev 0c)
fc:18.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: Misc control/status (rev 0c)
fc:19.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: REUT control/status (rev 0c)
fc:19.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: Misc control/status (rev 0c)
fc:1a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 8 (rev 0c)
fc:1b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 9 (rev 0c)
fd:00.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 1 (rev 0c)
fd:00.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 2 (rev 0c)
fd:00.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 3 (rev 0c)
fd:00.6 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 4 (rev 0c)
fd:01.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Power Controller (rev 0c)
fd:02.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 0 (rev 0c)
fd:03.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 1 (rev 0c)
fd:04.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 0 (rev 0c)
fd:05.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0a (rev 0c)
fd:05.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0b (rev 0c)
fd:05.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0c (rev 0c)
fd:05.6 Host bridge: Intel Corporation Device 2b53 (rev 0c)
fd:06.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 1 (rev 0c)
fd:07.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1a (rev 0c)
fd:07.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1b (rev 0c)
fd:07.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1c (rev 0c)
fd:07.6 Host bridge: Intel Corporation Device 2b5b (rev 0c)
fd:08.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 0 (rev 0c)
fd:09.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 1 (rev 0c)
fd:0a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 2 (rev 0c)
fd:0b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 3 (rev 0c)
fd:0c.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 4 (rev 0c)
fd:0d.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 5 (rev 0c)
fd:0e.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 6 (rev 0c)
fd:0f.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 7 (rev 0c)
fd:10.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 0-1 (rev 0c)
fd:10.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 2-3 (rev 0c)
fd:10.4 Host bridge: Intel Corporation Device 2b32 (rev 0c)
fd:10.6 Host bridge: Intel Corporation Device 2b3a (rev 0c)
fd:11.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 4-5 (rev 0c)
fd:11.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 6-7 (rev 0c)
fd:11.4 Host bridge: Intel Corporation Device 2b36 (rev 0c)
fd:11.6 Host bridge: Intel Corporation Device 2b3e (rev 0c)
fd:12.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 0 (rev 0c)
fd:13.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 1 (rev 0c)
fd:14.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: REUT control/status (rev 0c)
fd:14.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: Misc. control/status (rev 0c)
fd:15.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: REUT control/status (rev 0c)
fd:15.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: Misc. control/status (rev 0c)
fd:17.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: REUT control/status (rev 0c)
fd:17.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: Misc. control/status (rev 0c)
fd:18.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: REUT control/status (rev 0c)
fd:18.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: Misc control/status (rev 0c)
fd:19.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: REUT control/status (rev 0c)
fd:19.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: Misc control/status (rev 0c)
fd:1a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 8 (rev 0c)
fd:1b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 9 (rev 0c)
fe:00.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 1 (rev 0c)
fe:00.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 2 (rev 0c)
fe:00.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 3 (rev 0c)
fe:00.6 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 4 (rev 0c)
fe:01.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Power Controller (rev 0c)
fe:02.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 0 (rev 0c)
fe:03.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 1 (rev 0c)
fe:04.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 0 (rev 0c)
fe:05.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0a (rev 0c)
fe:05.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0b (rev 0c)
fe:05.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0c (rev 0c)
fe:05.6 Host bridge: Intel Corporation Device 2b53 (rev 0c)
fe:06.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 1 (rev 0c)
fe:07.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1a (rev 0c)
fe:07.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1b (rev 0c)
fe:07.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1c (rev 0c)
fe:07.6 Host bridge: Intel Corporation Device 2b5b (rev 0c)
fe:08.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 0 (rev 0c)
fe:09.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 1 (rev 0c)
fe:0a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 2 (rev 0c)
fe:0b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 3 (rev 0c)
fe:0c.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 4 (rev 0c)
fe:0d.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 5 (rev 0c)
fe:0e.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 6 (rev 0c)
fe:0f.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 7 (rev 0c)
fe:10.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 0-1 (rev 0c)
fe:10.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 2-3 (rev 0c)
fe:10.4 Host bridge: Intel Corporation Device 2b32 (rev 0c)
fe:10.6 Host bridge: Intel Corporation Device 2b3a (rev 0c)
fe:11.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 4-5 (rev 0c)
fe:11.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 6-7 (rev 0c)
fe:11.4 Host bridge: Intel Corporation Device 2b36 (rev 0c)
fe:11.6 Host bridge: Intel Corporation Device 2b3e (rev 0c)
fe:12.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 0 (rev 0c)
fe:13.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 1 (rev 0c)
fe:14.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: REUT control/status (rev 0c)
fe:14.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: Misc. control/status (rev 0c)
fe:15.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: REUT control/status (rev 0c)
fe:15.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: Misc. control/status (rev 0c)
fe:16.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 2: REUT control/status (rev 0c)
fe:16.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 2: Misc. control/status (rev 0c)
fe:17.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: REUT control/status (rev 0c)
fe:17.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: Misc. control/status (rev 0c)
fe:18.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: REUT control/status (rev 0c)
fe:18.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: Misc control/status (rev 0c)
fe:19.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: REUT control/status (rev 0c)
fe:19.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: Misc control/status (rev 0c)
fe:1a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 8 (rev 0c)
fe:1b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 9 (rev 0c)
ff:00.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 1 (rev 0c)
ff:00.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 2 (rev 0c)
ff:00.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 3 (rev 0c)
ff:00.6 Host bridge: Intel Corporation Xeon Processor E7 Product Family System Configuration Controller 4 (rev 0c)
ff:01.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Power Controller (rev 0c)
ff:02.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 0 (rev 0c)
ff:03.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Caching Agent 1 (rev 0c)
ff:04.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 0 (rev 0c)
ff:05.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0a (rev 0c)
ff:05.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0b (rev 0c)
ff:05.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 0c (rev 0c)
ff:05.6 Host bridge: Intel Corporation Device 2b53 (rev 0c)
ff:06.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Home Agent 1 (rev 0c)
ff:07.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1a (rev 0c)
ff:07.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1b (rev 0c)
ff:07.4 Host bridge: Intel Corporation Xeon Processor E7 Product Family Memory Controller 1c (rev 0c)
ff:07.6 Host bridge: Intel Corporation Device 2b5b (rev 0c)
ff:08.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 0 (rev 0c)
ff:09.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 1 (rev 0c)
ff:0a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 2 (rev 0c)
ff:0b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 3 (rev 0c)
ff:0c.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 4 (rev 0c)
ff:0d.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 5 (rev 0c)
ff:0e.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 6 (rev 0c)
ff:0f.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 7 (rev 0c)
ff:10.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 0-1 (rev 0c)
ff:10.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 2-3 (rev 0c)
ff:10.4 Host bridge: Intel Corporation Device 2b32 (rev 0c)
ff:10.6 Host bridge: Intel Corporation Device 2b3a (rev 0c)
ff:11.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 4-5 (rev 0c)
ff:11.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Router Port 6-7 (rev 0c)
ff:11.4 Host bridge: Intel Corporation Device 2b36 (rev 0c)
ff:11.6 Host bridge: Intel Corporation Device 2b3e (rev 0c)
ff:12.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 0 (rev 0c)
ff:13.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Test and Debug 1 (rev 0c)
ff:14.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: REUT control/status (rev 0c)
ff:14.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 0: Misc. control/status (rev 0c)
ff:15.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: REUT control/status (rev 0c)
ff:15.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 1: Misc. control/status (rev 0c)
ff:16.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 2: REUT control/status (rev 0c)
ff:16.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 2: Misc. control/status (rev 0c)
ff:17.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: REUT control/status (rev 0c)
ff:17.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family QPI Physical Port 3: Misc. control/status (rev 0c)
ff:18.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: REUT control/status (rev 0c)
ff:18.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 0: Misc control/status (rev 0c)
ff:19.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: REUT control/status (rev 0c)
ff:19.2 Host bridge: Intel Corporation Xeon Processor E7 Product Family SMI Physical Port 1: Misc control/status (rev 0c)
ff:1a.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 8 (rev 0c)
ff:1b.0 Host bridge: Intel Corporation Xeon Processor E7 Product Family Last Level Cache Coherence Engine 9 (rev 0c)

// update

• this also happened with allot java jobs running, the filesystem felt "slow" before the freeze. (ls -l, cat, ...)
• changed the title to reflect workload, and not only tar.

We received a quote for the following server setup :

• Intel Xeon E5-2609V2
• 2 x WD RE 2TB SATA III
• 34 x WD RE 4TB SAS II
• 32 GB memory (ECC)
• LSI MegaRAID 9271-8i bulk, 8 x SATA III/SAS II internal hardware RAID
• LSI CacheVault voor 9266-9271 series

We wanne add (directly) a JBOD to that server, half filled with 8TB drives, we can extend later. They suggested :

• LSI MegaRAID 9380-8e
• 22 x HGST Ultrastar 8TB He8 enterprise, SAS III

Now this was based on our previous server, which we setup as a ZFS server and did not have much "pleasure" from. (although the configuration was to blame I guess)

I have a few questions about this setup : - The argument to take 2x2TB is, use it as mirror for the system, since when a disk has to be replaced IO is sluggish during rebuild. Speed is not our real problem, space is, also we have a online backup, that will be only used as read platform (during problems). Would 36 x 4TB be a better choice ? (36 = 3*12 disk in a pool) - is 32 Gb memory enough ? (ZFS on linux, taking in consideration the JBOD at max capacity 44*8+32*4) - This is a raid controller, a JBOD/HBA (?) would be a better choice ? If so, what kind of JBOD should I be looking for ? - How would I best setup this system to be "ready" for upgrading the next 22 disks in the JBOD ? (its a 44 disk JBOD, 22 slots are filled)

Some more info based on the comments :

• uptime/availability: we don't care if it drops out for a few minutes, aslong as this is not all to common. no HA needed. In essence this will be a hot backup for our current storage server. We mainly read and writing is not speed limited. (by far)
• Reading speed is important but we don't want to give up space for it
• Write speed is not that important, mostly its streams from machine while large files are written there copy's so it can run overnight.

Our storage server has some problems, just ago we had a broken disk (WD 4TB RE SAS), though the raidcontroller (LSI MegaRAID 9271-8i) kept the disk online (status : ok). Only Media error showed 1 error. We decided to be save and replace the disk, during the resilvering a second and a tirth disk where marked as (resilvering) although only one disk showed 1 read error. Today the resilver is complete (no corruption, all oke), I started a scrub and was met with this :

zpool status
  pool: data
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub in progress since Thu Jan 14 10:50:00 2016
    2.71T scanned out of 111T at 718M/s, 43h59m to go
    0 repaired, 2.44% done
config:

        NAME                     STATE     READ WRITE CKSUM
        data                     DEGRADED     0     0     0
          raidz2-0               DEGRADED     0     0     0
            br0c2                ONLINE       0     0     0
            br1c2                ONLINE       0     0     0
            br2c2                ONLINE       0     0     0
            br0c3                ONLINE       0     0     0
            br1c3                ONLINE       0     0     0
            br2c3                ONLINE       0     0     0
            r2c1                 ONLINE       0     0     0
            r1c2                 ONLINE       0     0     0
            r5c3                 ONLINE       0     0     0
            sdb                  ONLINE       0     0     0
            sdc                  ONLINE       0     0     0
            7196084230607724634  FAULTED      0     0     0  was /dev/sdai1
            r5c0                 ONLINE       0     0     0
            r0c1                 ONLINE       0     0     0
            r1c1                 ONLINE       0     0     0
            r3c1                 ONLINE       0     0     0
            r4c1                 ONLINE       0     0     0
          raidz2-1               ONLINE       0     0     0
            r5c1                 ONLINE       0     0     0
            r0c2                 ONLINE       0     0     0
            r2c2                 ONLINE       0     0     0
            r3c2                 ONLINE       0     0     0
            r4c2                 ONLINE       0     0     0
            r5c2                 ONLINE       0     0     0
            r0c3                 ONLINE       0     0     0
            r1c3                 ONLINE       0     0     0
            r2c3                 ONLINE       0     0     0
            r3c3                 ONLINE       0     0     0
            r4c3                 ONLINE       0     0     0
            br0c0                ONLINE       0     0     0
            br1c0                ONLINE       0     0     0
            br2c0                ONLINE       0     0     0
            br0c1                ONLINE       0     0     0
            br1c1                ONLINE       0     0     0
            br2c1                ONLINE       0     0     0

errors: No known data errors

/dev/sdai1 is online and the raidcontroller is not showing any error (not even media error). Can I try and put the disk online / offline ?

update : I try'd to deatach the disk but it refuses to do so, I was under to impression that raidz2-0 has 2 parity disks (and raidz2-1 also 2) so why can't I detach ?

zpool detach data 7196084230607724634
cannot detach 7196084230607724634: only applicable to mirror and replacing vdevs

We have a VPS (2GB RAM/2 vCPU's). That is serving a phpBB forum on PHP 5.6, Centos 6, Apache 2.2. However sometimes the forum is super slow to respond. Using Chrome network browser I found that TTFB sometimes goes up to 30 seconds. Weirdly enough also cached items have this problem. (see screenshot)

I already tried :

• load of the server during the issue is not above .20, and there is free memory.

• I pinged the server : around 10ms

• install zend opcache (99.9% hits)

• install memcached (stats)

• I installed mytop to see if MySQL is the problem but as far as I understand MySQL is not doing much

  Queries: 6.0M qps: 26 Slow: 4.0 Se/In/Up/De(%): 75/02/08/01 qps now: 2 Slow qps: 0.0 Threads: 1 ( 1/ 0) 00/00/00/00 Key Efficiency: 99.4% Bps in/out: 4.4k/51.1k Now in/out: 48.4/ 7.5k

Since its not always the case, the only way to replicate this issue, is spamming F5, although I'm not sure that is really replicating the problem. I could in lesser extend (2s ttfb) replicate this with the same page copy'd in html (removed the dynamic part).

http://s21.postimg.org/ql6eywn3b/ttfb.png

I have keepalive on (100 max, 15 seconds), with maxclients 30. Where do I go from here ?

We have problems with our storage server, it was not updated for a while so we tried to updated it, tho during the update process (it needed some 2k package updates) it totally blacked out. (no screen on the motherboard, ssh dead, but still some io on the disks: blinking leds)

We rebooted the machines to see whats going on. The boot failed and we got this error :

Server fails to boot with lvm error : ¨Invalid argument for --available: ay Error during parsing of command line.¨

The only solution I could find is on redhat knowledge bank and is protected by a paywall :(

I have verified that all the data on the root (ext4) and home (ext4) are both oke and can be accessed using a rescue disk. I could also mount these and gain access. So I'm pretty sure this is a configuration error. I have however no idea how to fix it.

This is running on a centos 6 machine, its saying (etc/redhat-release) that its 6.7 but I think its older, and that 6.7 is the upgraded version.

I'm not sure if its usefull in this context but I dumped (grep -v '#') the config /etc/lvm/lvm.conf

So ZFS is reporting some "read issues", so it would seem that this disk is failing, based on the fact nothing given in the ZFS-8000-9P document reports has occurred we are aware of. These disks are fairly new, the only issue we had recently was a full ZFS.

The ZFS runs on top of a LSI MegaRAID 9271-8i, all disks run "raid 0" per disk. I am not very familiar with this raid card, so I found a script that returns data derived from the megacli command line tool. I added 1 drive to show the setup, they are all setup the same. (system disks are different)

zpool status output

  pool: data
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            br0c2   ONLINE       0     0     0
            br1c2   ONLINE       0     0     0
            br2c2   ONLINE       0     0     0
            br0c3   ONLINE       0     0     0
            br1c3   ONLINE       0     0     0
            br2c3   ONLINE       0     0     0
            r2c1    ONLINE       0     0     0
            r1c2    ONLINE       0     0     0
            r5c3    ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdf     ONLINE       0     0     0
            sdg     ONLINE       0     0     0
            r3c1    ONLINE       0     0     0
            r4c1    ONLINE       2     0     0
... cut raidz2-1 ...
errors: No known data errors

The output of LSI script

Virtual Drive: 32 (Target Id: 32)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 3.637 TB
Sector Size         : 512
Is VD emulated      : No
Parity Size         : 0
State               : Optimal
Strip Size          : 512 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
PI type: No PI

Is VD Cached: No

The script doesn't report any faulty disk, nor does the raidcontroller mark the drive as faulty. I found some other topics zpool error that gave the advice to clear the error and run a scrub. Now my question is, when is the threshold to run a scrub, how long would this take (assuming this zfs raid will take a performance hit for running scrub) Also when this disk is really fautly, will hot-swapping initialize a "rebuild" ? All the disks are "Western Digital RE 4TB, SAS II, 32MB, 7200rpm, enterprise 24/7/365". Is there a system that will check for zfs errors, since this was just a routine manual check ?

zfs version : 0.6.4.1 zfsonlinux

I know 2 read errors are not allot, but i'd prefer to be replacing disks to early then to late.

To sketch the situation :

• a bunch of servers (~15 going from 4u to 1u, some redudant PSU some single)
• 2 UPS (1 APC I can read using apcupsd)
• 2 PDU

So I came upon this situation, and I honestly have no idea what the best solution would be. We have currently put most of the servers (even redundant PSU) on 1 UPS (older one, no APC) Then when we started drawing too much juice from it, it started to complain and they bought another one, and changed the probably most power hungry servers (clusters) on the new UPS (APC)

I don't mind servers going offline during a power break, and I would like them to go down well before the UPS give out. Currently the servers run until they don't get any more power, then die painfully.

So the reasonable situation would be that we dedup the servers who can so that they pull 50/50 from each UPS (as far as possible). However, during a power down, one of the batteries would be finished before the other one, this would result that ALL the servers draw juice from the other UPS, probably pulling way to much power and resulting in a instant power break?

So my questions are do PSU pull 50% from each power source? And how can I (if necessary with new hardware) fix this situation. And what is the worst that can happen when you pull too much from a UPS ?

I want to protect the servers from high peaks in power and in short outages of <5 min. Anything beyond this time-frame is explainable up the ladder. Or servers are configured OK and we know they can power cycle if deemed necessary.