AlexD's questions

Yesterday one of the disks in RAID 1 mirror attached to Adaptec 5405 died and was replaced by new one but after about 2 hours of rebuilding array went offline. Hoster staff updated firmware on controller and forced array online back. System booted normally and arcconf getconfig 1 showed following output:

Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status                        : Optimal
Channel description                      : SAS/SATA
Controller Model                         : Adaptec 5405
Controller Serial Number                 : 1D091194BC6
Physical Slot                            : 18
Temperature                              : 85 C/ 185 F (Normal)
Installed memory                         : 256 MB
Copyback                                 : Disabled
Background consistency check             : Disabled
Automatic Failover                       : Enabled
Global task priority                     : High
Performance Mode                         : Default/Dynamic
 Stayawake period                         : Disabled
Spinup limit internal drives             : 0
Spinup limit external drives             : 0
Defunct disk drive count                 : 0
Logical devices/Failed/Degraded          : 1/0/1
SSDs assigned to MaxCache pool           : 0
Maximum SSDs allowed in MaxCache pool    : 8
MaxCache Read Cache Pool Size            : 0.000 GB
MaxCache flush and fetch rate            : 0
MaxCache Read, Write Balance Factor      : 3,1
NCQ status                               : Enabled
Statistics data collection mode          : Enabled
--------------------------------------------------------
Controller Version Information
--------------------------------------------------------
BIOS                                     : 5.2-0 (18948)
Firmware                                 : 5.2-0 (18948)
Driver                                   : 1.1-7 (28000)
Boot Flash                               : 5.2-0 (18948)
--------------------------------------------------------
Controller Battery Information
--------------------------------------------------------
Status                                   : Not Installed 

----------------------------------------------------------------------
Logical device information
----------------------------------------------------------------------
Logical device number 0
Logical device name                      : 
RAID level                               : 1
Status of logical device                 : Degraded
Size                                     : 953334 MB
Read-cache mode                          : Enabled
MaxCache preferred read cache setting    : Enabled
Write-cache mode                         : Disabled (write-through)
Write-cache setting                      : Disabled (write-through)
Partitioned                              : Yes
Protected by Hot-Spare                   : No
Bootable                                 : Yes
Failed stripes                           : No
Power settings                           : Disabled
--------------------------------------------------------
Logical device segment information
--------------------------------------------------------
Segment 0                                : Present (Controller:1,Connector:0,Device:1)             9VP3AGB1
Segment 1                                : Rebuilding (Controller:1,Connector:0,Device:0)             Z1D49LKS


----------------------------------------------------------------------
Physical Device information
----------------------------------------------------------------------
  Device #0
     Device is a Hard drive
     State                              : Rebuilding
     Supported                          : Yes
     Transfer Speed                     : SATA 3.0 Gb/s
     Reported Channel,Device(T:L)       : 0,0(0:0)
     Reported Location                  : Connector 0, Device 0
     Vendor                             : 
     Model                              : ST1000DM003-9YN1
     Firmware                           : CC4H
     Serial number                      : Z1D49LKS
     Size                               : 953869 MB
     Write Cache                        : Enabled (write-back)
     FRU                                : None
     S.M.A.R.T.                         : No
     S.M.A.R.T. warnings                : 0
     Power State                        : Full rpm
     Supported Power States             : Full rpm,Powered off,Reduced rpm
     SSD                                : No
     MaxCache Capable                   : No
     MaxCache Assigned                  : No
     NCQ status                         : Enabled
  Device #1
     Device is a Hard drive
     State                              : Online
     Supported                          : Yes
     Transfer Speed                     : SATA 3.0 Gb/s
     Reported Channel,Device(T:L)       : 0,1(1:0)
     Reported Location                  : Connector 0, Device 1
     Vendor                             : 
     Model                              : ST31000528AS
     Firmware                           : CC38
     Serial number                      : 9VP3AGB1
     Size                               : 953869 MB
     Write Cache                        : Enabled (write-back)
     FRU                                : None
     S.M.A.R.T.                         : No
     S.M.A.R.T. warnings                : 0
     Power State                        : Full rpm
     Supported Power States             : Full rpm,Powered off
     SSD                                : No
     MaxCache Capable                   : No
     MaxCache Assigned                  : No
     NCQ status                         : Enabled

and aacraid-status

-- Controller informations --
-- ID | Model | Status
c0 | Adaptec 5405 | Optimal

-- Arrays informations --
-- ID | Type | Size | Status | Task | Progress
c0u0 | RAID1 | 953G | Degraded | Rebuild | 44%

-- Disks informations
-- ID | Model | Status

There is at least one disk/array in a NOT OPTIMAL state.

smartctl for drive #1 showed that it is in pre-failure state:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   114   099   006    Pre-fail  Always       -       69962026
  3 Spin_Up_Time            0x0003   095   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -        93
  5 Reallocated_Sector_Ct   0x0033   002   002   036    Pre-fail  Always   FAILING_NOW  4015
  7 Seek_Error_Rate         0x000f   083   060   030    Pre-fail  Always       -       222073391
  9 Power_On_Hours          0x0032   073   073   000    Old_age   Always       -       24485
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       72
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   006   006   000    Old_age   Always       -       94
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       1507556589923
189 High_Fly_Writes         0x003a   095   095   000    Old_age   Always       -       5
190 Airflow_Temperature_Cel 0x0022   059   051   045    Old_age   Always       -       41 (Min/Max 40/41)
194 Temperature_Celsius     0x0022   041   049   000    Old_age   Always       -       41 (0 19 0 0)
195 Hardware_ECC_Recovered  0x001a   045   028   000    Old_age   Always       -       69962026
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       202082506249469
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1020514404
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       590957029

and array failed to rebuild again and went offline.

What is the best way to proceed now with getting this system back online?

I'm running LXC container (lxc 0.7.2-1) on Debian squeeze with bridge networking configured by the book with public IP and I have problem that it exposes fake MAC address from LXC container to external network where it gets blocked by switch port security so preventing LXC container from communicating with the world. I run another LXC container on Ubuntu 12.04.2 LTS (lxc 0.7.5-3ubuntu67) with same network configuration but I don't have any problems as it doesn't expose fake MACs to external network and all communications goes with host's MAC.

Currently I'm trying to find difference in network configuration between hosts but without much success. Could it be possibly version dependent behavior of lxc itself?

Debian /etc/network/interfaces

auto br0
iface br0 inet static
    bridge_ports eth0
    bridge_fd 0
    bridge_stp off
    bridge_maxwait 0
    address   y.y.y.9
    netmask   255.255.255.192
    broadcast y.y.y.63
    gateway   y.y.y.1
    pre-up iptables-restore < /etc/iptables.up.rules

Debian LXC network config

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = fe:95:57:4b:b4:9b
lxc.network.ipv4 = y.y.y.12/26

Ubuntu /etc/network/interfaces

auto br0
iface br0 inet static
    bridge_ports eth0
    bridge_fd 0
    bridge_stp off
    bridge_maxwait 0
    address   z.z.z.146
    netmask   255.255.255.0
    broadcast z.z.z.255
    gateway   z.z.z.1
    pre-up iptables-restore < /etc/iptables.rules
    up route add x.x.x.1 br0

Ubuntu LXC network config

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.ipv4 = x.x.x.1/32
lxc.network.hwaddr = 00:16:3e:87:b5:b9

One of my servers shows strange change in its monitored parameters. Last Sunday evening one of voltage measures ( in0 which I think is Vcore) suddenly increased by 20% from ~1V to ~1.20V on average. There is corresponding significant increase in reported CPU core temperatures and CPU fan speed but very slight increase in CPU frequency (1.20GHz -> 1.25GHz) and CPU base load.

What could this mean? I suspect failing PSU or MB. Is there any other good explanations for such behavior? There were no changes of server hardware, BIOS setting or software at this point of time.

Server MB is Gigabyte GA-MA69VM-S2 with AMD Athlon(tm) Dual Core Processor 4450e. OS is Ubuntu 12.04.2 LTS with kernel 3.2.0-45-generic. Monitoring is done by lm-sensors 1:3.3.1-2ubuntu1. Current output from sensors command is below graphs.

k8temp-pci-00c3
Adapter: PCI adapter
Core0 Temp:   +38.0°C  
Core0 Temp:   +35.0°C  
Core1 Temp:   +35.0°C  
Core1 Temp:   +37.0°C  

it8716-isa-0228
Adapter: ISA adapter
in0:          +1.20 V  (min =  +0.00 V, max =  +4.08 V)
in1:          +1.89 V  (min =  +0.00 V, max =  +4.08 V)
in2:          +3.36 V  (min =  +0.00 V, max =  +4.08 V)
+5V:          +2.98 V  (min =  +0.00 V, max =  +4.08 V)
in4:          +3.07 V  (min =  +0.00 V, max =  +4.08 V)
in5:          +3.23 V  (min =  +0.00 V, max =  +4.08 V)
in6:          +0.10 V  (min =  +0.00 V, max =  +4.08 V)
5VSB:         +3.02 V  (min =  +0.00 V, max =  +4.08 V)
Vbat:         +3.02 V  
fan1:        1555 RPM  (min =    0 RPM)
fan2:        2860 RPM  (min =    0 RPM)
temp1:        +29.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
temp2:        +34.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
temp3:        +31.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermal diode
cpu0_vid:    +1.200 V
intrusion0:  ALARM

Yesterday I got report about SSD ATA errors on one of my hosts.

SSD disk is 128MB OCZ-VERTEX4 Firmware rev 1.3 about 8 months old.

OS is Ubuntu 11.04 running kernel 2.6.38-16-generic.

Motherboard is Intel DP35DP.

There are no read errors or any other disk errors since these two below.

Should I prepare replacement drive?

Smart attributes:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0000   ---   ---   ---    Old_age   Offline      -       393222
  3 Spin_Up_Time            0x0000   100   100   000    Old_age   Offline      -       0
  4 Start_Stop_Count        0x0000   100   100   000    Old_age   Offline      -       0
  5 Reallocated_Sector_Ct   0x0000   100   100   000    Old_age   Offline      -       0
  9 Power_On_Hours          0x0000   ---   ---   ---    Old_age   Offline      -       507536484
 12 Power_Cycle_Count       0x0000   ---   ---   ---    Old_age   Offline      -       1664100
232 Available_Reservd_Space 0x0000   100   100   000    Old_age   Offline      -       4804710657
233 Media_Wearout_Indicator 0x0000   099   000   000    Old_age   Offline      -       99

Kernel log:

Jun  1 11:50:42 kernel: [424453.095411] ata4: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen
Jun  1 11:50:42 kernel: [424453.095415] ata4: irq_stat 0x00400040, connection status changed
Jun  1 11:50:42 kernel: [424453.095418] ata4: SError: { PHYRdyChg DevExch }
Jun  1 11:50:42 kernel: [424453.095422] ata4: hard resetting link
Jun  1 11:50:42 kernel: [424453.840022] ata4: SATA link down (SStatus 0 SControl 300)
Jun  1 11:50:44 kernel: [424455.948532] ata4: hard resetting link
Jun  1 11:50:45 kernel: [424456.490021] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun  1 11:50:45 kernel: [424456.490288] ata4.00: configured for UDMA/133
Jun  1 11:50:45 kernel: [424456.490294] ata4: EH complete

Jun  1 19:18:23 kernel: [451311.319525] ata4.00: exception Emask 0x10 SAct 0x0 SErr 0x4010000 action 0xe frozen
Jun  1 19:18:23 kernel: [451311.319529] ata4.00: irq_stat 0x00400040, connection status changed
Jun  1 19:18:23 kernel: [451311.319532] ata4: SError: { PHYRdyChg DevExch }
Jun  1 19:18:23 kernel: [451311.319535] ata4.00: failed command: FLUSH CACHE
Jun  1 19:18:23 kernel: [451311.319541] ata4.00: cmd e7/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Jun  1 19:18:23 kernel: [451311.319542]          res 40/00:0c:78:c6:c7/00:00:0a:00:00/40 Emask 0x10 (ATA bus error)
Jun  1 19:18:23 kernel: [451311.319545] ata4.00: status: { DRDY }
Jun  1 19:18:23 kernel: [451311.319549] ata4: hard resetting link
Jun  1 19:18:23 kernel: [451312.060033] ata4: SATA link down (SStatus 0 SControl 300)
Jun  1 19:18:23 kernel: [451314.082062] ata4: hard resetting link
Jun  1 19:18:23 kernel: [451314.630022] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun  1 19:18:23 kernel: [451314.630295] ata4.00: configured for UDMA/133
Jun  1 19:18:23 kernel: [451314.630298] ata4.00: retrying FLUSH 0xe7 Emask 0x10
Jun  1 19:18:23 kernel: [451314.630320] ata4: EH complete

I'm trying to implement puppet /etc/hosts handling using hiera as data source, but my current implementation requires data duplication because I could pass only array of strings as $name to host resource.

Is there any way to avoid such data duplication? Or maybe it is possible to use templates but keep ability to have local modifications to /etc/hosts file?

class hosts ($hosts = hiera("hosts"), $hostsdefs = hiera("hostsdefs"))  { 

  define hostentry( ) { 
    host{ $name: ip => $hostsdefs[$name][ipaddress], 
                 host_aliases => $hostsdefs[$name][host_aliases]  } 
  } 

  hostentry{ $hosts: } 
} 

YAML data for hiera:

---- 
hosts: 
  - host1.example.com 
  - host2.example.com 
hostsdefs: 
  host1.example.com: 
    ipaddress: 10.0.0.1 
    host_aliases: host1 
 host2.example.com: 
    ipaddress: 10.0.0.2 
    host_aliases: host2 

Currently I have a few FreeBSD servers with one of them acting as build box sharing its /usr/ports and /usr/ports/distfiles (as separate file systems) through NFS. Recently I've added OpenBSD and Gentoo boxes to this network. Now I want to use existing FreeBSD build box to share OpenBSD and Gentoo distfiles through NFS. Is it safe to place all distfiles for different systems into the same directory?

I have a few dozen servers behind OpenBSD firewall with port forwarding. Most rules are very similar and differ only in IPs or sometimes in ports forwarded, so I want to compact them to remove excessive repetition but I've found that it is impossible to use tables with rdr-to rules. Is there any way to improve this configuration? May be there is option to use pf macros to generate multiple rules at once? I can't use external preprocessor at the moment.

Example set of rules:

pass in  on $extif proto tcp from any     to 10.0.0.213   port {25,80,443} rdr-to 172.16.1.193
pass in  on $intif proto tcp from $intnet to 10.0.0.213   port {25,80,443} rdr-to 172.16.1.193
pass out on $intif proto tcp from any     to 172.16.1.193 port {25,80,443} received-on $intif nat-to $intif

pass in  on $extif proto tcp from any     to 10.0.0.214   port {25,80,443} rdr-to 172.16.1.194
pass in  on $intif proto tcp from $intnet to 10.0.0.214   port {25,80,443} rdr-to 172.16.1.194
pass out on $intif proto tcp from any     to 172.16.1.194 port {25,80,443} received-on $intif nat-to $intif

pass in  on $extif proto tcp from any     to 10.0.0.215   port {25,80,443,3389} rdr-to 172.16.1.195
pass in  on $intif proto tcp from $intnet to 10.0.0.215   port {25,80,443,3389} rdr-to 172.16.1.195
pass out on $intif proto tcp from any     to 172.16.1.195 port {25,80,443,3389} received-on $intif nat-to $intif