Andy Shinn's questions

I have connection draining enabled for a ELB. According to the documentation at http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#conn-drain, a deregistering or unhealthy instance will keep existing connections open:

Connection draining causes the ELB load balancer to stop sending new requests to a deregistering instance or an unhealthy instance, while keeping the existing connections open.

I'd like to perform some maintenance on EC2 nodes in the ELB. It would be nice to gracefully take them out of rotation manually. I have tried failing the health checks and manually removing the EC2 instance from the ELB. Both of these actions kill existing connections to the instance.

How can I manually set a EC2 instance to deregistering or unhealthy state so that it will continue to serve existing connections but not accept new ones?

I am using CoreOS to schedule systemd units with fleet. I have two units (firehose.service and firehose-announce.service. I am trying to get the firehose-announce.service to start and stop along with the firehose.service. Here is the unit file for firehose-announce.service:

[Unit]
Description=Firehose etcd announcer
BindsTo=firehose@%i.service
After=firehose@%i.service
Requires=firehose@%i.service

[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=30s
ExecStartPre=/bin/sh -c 'sleep 1'
ExecStart=/bin/sh -c "port=$(docker inspect -f '{{range $i, $e := .NetworkSettings.Ports }}{{$p := index $e 0}}{{$p.HostPort}}{{end}}' firehose-%i); echo -n \"Adding socket $COREOS_PRIVATE_IPV4:$port/tcp to /firehose/upstream/firehose-%i\"; while netstat -lnt | grep :$port >/dev/null; do etcdctl set /firehose/upstream/firehose-%i $COREOS_PRIVATE_IPV4:$port --ttl 300 >/dev/null; sleep 200; done"
RestartSec=30s
Restart=on-failure

[X-Fleet]
X-ConditionMachineOf=firehose@%i.service

I am trying to use BindsTo with the notion that start and stop of firehose.service will also start or stop firehose-announce.service. But this never happens correctly. If firehose.service is stopped, then firehose-announce.service goes to failed state. But when I start firehose.service, the firehose-announce.service doesn't start up.

What am I doing wrong here?

I am looking to increase storage of two RDS instances (just the storage space allocated, not the instance type or other parameters). The documentation at https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.ModifyingExisting suggests:

You can change from standard storage to Provisioned IOPS storage, or from Provisioned IOPS to standard storage, as well as increase storage, with little to no downtime.

I would definitely schedule a maintenance window before performing the change. But the documentation seems a little vague in this area. For someone who might have done this before, what is "little to no downtime"? Can I expect 5 seconds or is it more like 5 minutes?

Update July, 2019:

I've updated the link to the correct and updated AWS documentation (which was broken). The newer documentation has a blurb that helps answer the original question as well:

In most cases, scaling storage doesn't require any outage and doesn't degrade performance of the server. After you modify the storage size for a DB instance, the status of the DB instance is Storage-optimization. The DB instance is fully operational after a storage modification. However, you can't make further storage modifications either for six hours or while the DB instance status is storage-optimization, whichever is longer.

However, a special case is if you have a SQL Server DB instance and haven't modified the storage configuration since November 2017. In this case, you might experience a short outage of a few minutes when you modify your DB instance to increase the allocated storage. After the outage, the DB instance is online but in the Storage-optimization state. Performance might be degraded during storage optimization.

I'm having trouble figuring out how to remove systemd units that no longer have files. They still seem to linger in the system somehow.

The old broken units I am trying to remove:

core@ip-172-16-32-83 ~ $ systemctl list-units --all firehose-router*
  UNIT                       LOAD      ACTIVE SUB    DESCRIPTION
<E2><97><8F> [email protected] not-found failed failed [email protected]
<E2><97><8F> [email protected] not-found failed failed [email protected]

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.

The files do not exist, yet a reload still has these units lingering:

core@ip-172-16-32-83 ~ $ systemctl list-unit-files [email protected]
core@ip-172-16-32-83 ~ $ sudo systemctl daemon-reload
core@ip-172-16-32-83 ~ $ systemctl list-units --all firehose-router*
  UNIT                       LOAD      ACTIVE SUB    DESCRIPTION
<E2><97><8F> [email protected] not-found failed failed [email protected]
<E2><97><8F> [email protected] not-found failed failed [email protected]

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.

There are no files related to them that I can find:

core@ip-172-16-32-83 ~ $ sudo find /var/run/systemd -name "*firehose-router*"
core@ip-172-16-32-83 ~ $ find /etc/systemd/ -name "*firehose-router*"
core@ip-172-16-32-83 ~ $ find /usr/lib/systemd/ -name "*firehose-router*"
core@ip-172-16-32-83 ~ $

So how do I get rid of these?

I'm looking for a way to set CoreOS sysctl settings during its cloud-init stage.

The CoreOS version of cloud-init only allows for a handful of configuration directives and is not the same as the regular cloud-init. For example, there is no runcmd section (see http://coreos.com/docs/cluster-management/setup/cloudinit-cloud-config/).

systemd provides a way to manage sysctl settings though files (http://www.freedesktop.org/software/systemd/man/sysctl.d.html). I am using the CoreOS cloud-init write_files section to create a file at /etc/sysctl.d/50-nf_conntrack.conf. But it won't be picked up because the CoreOS cloud configuration happens after the sysctl.d configuration has already taken place.

Perhaps I could somehow use another systemd unit file to restart the sysctl.d unit? How could this be accomplished?

I'm trying to setup a simple virtual server in Keepalived and when the servers get removed from the pool, they do not get re-added automatically. In the Keepalived server syslog I see:

Oct 01 11:37:39 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:40 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:40 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:41 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:44 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:47 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:48 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:48 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:49 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:49 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:50 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:50 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:51 lb2 kernel:  IPVS: WRR: no destination available: no destinations present
Oct 01 11:37:51 lb2 kernel:  IPVS: WRR: no destination available: no destinations present

ipvsadm outputs:

[ashinn@lb2 ~]$ sudo ipvsadm

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  700000 wrr persistent 300

Here is the relevant part from the keepalived.conf:

virtual_server fwmark 700000 {
    delay_loop 20
    lb_algo wrr
    lb_kind NAT
    persistence_timeout 300
    protocol TCP

    real_server 10.10.35.11 443 {
        weight 24
        SSL_GET {
            url {
                path /check.php
            }
        }
    }

    real_server 10.10.35.12 443 {
        weight 24
        SSL_GET {
            url {
                path /check.php
            }
        }
    }
}

The Keepalived server can indeed reach the real servers:

[ashinn@lb2 ~]$ curl -k https://10.10.35.11/check.php
FAIL<br/>
/usr/bin/mencoder is not executable.<br/>
dio extension is not loaded.<br/>

Any idea why the servers don't get re-added after they are removed?

I'm getting the following in my Puppet master syslog over and over:

Sep 27 11:52:05 puppet1 puppet-master:  Not collecting exported resources without storeconfigs
Sep 27 11:52:06 puppet1 puppet-master:  Not collecting exported resources without storeconfigs
Sep 27 11:52:06 puppet1 puppet-master:  Not collecting exported resources without storeconfigs

I'm not actually using storeconfigs:

[ashinn@puppet1 ~]$ cat /etc/puppet/puppet.conf
[agent]
server                    = puppet.mydomain.com
environment               = production
report                    = true

[main]
logdir                    = /var/log/puppet
vardir                    = /var/lib/puppet
ssldir                    = /var/lib/puppet/ssl
rundir                    = /var/run/puppet
factpath                  = $vardir/lib/facter
pluginsync                = true
certname                  = puppet1.mydomain.com

[master]
modulepath                = $confdir/environments/$environment/modules
manifest                  = $confdir/environments/$environment/manifests/site.pp
templatedir               = $confdir/templates
autosign                  = $confdir/autosign.conf
ssl_client_header         = SSL_CLIENT_S_DN
ssl_client_verify_header  = SSL_CLIENT_VERIFY
report                    = true
reports                   = hipchat

Any way I can suppress these messages? What do they actually come from?

I'd like to setup Vagrant to use my current Puppet master as a provisioner. But I don't like the idea of embedding the SSL key pair in the .box file.

What I am currently doing is distributing the ca.pem, public, private, and cert key files for the Puppet agent with the Vagrant file and using this shell script to bootstrap the Puppet agent:

$puppet_ssl = <<SCRIPT
sudo mkdir -p /var/lib/puppet/ssl/{certs,private_keys,public_keys}
sudo chown puppet /var/lib/puppet/ssl/{certs,private_keys,public_keys}
sudo cp /vagrant/puppet/certs/appdev.mydomain.com.pem /vagrant/puppet/certs/ca.pem /var/lib/puppet/ssl/certs/
sudo cp /vagrant/puppet/private_keys/appdev.mydomain.com.pem /var/lib/puppet/ssl/private_keys/
sudo cp /vagrant/puppet/public_keys/appdev.mydomain.com.pem /var/lib/puppet/ssl/public_keys/
SCRIPT

This seems to work OK if the developer has the files and everything is in the correct directories.

My question: Is there a better way to distribute arbitrary files that need to be pre-loaded onto the Vagrant VM without embedding them into the .box file?

I am receiving the following error using a puppet module on my Puppet master to manage both master (Apache / Passenger) and agent

[ashinn@puppet1 ~]$ sudo puppet agent --test
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/os_maj_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass puppet_server to Class[Puppet::Agent] at /etc/puppet/environments/production/modules/puppet/manifests/init.pp:181 on node puppet1.mydomain.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

The node definition for puppet1.mydomain.com is pretty simple:

node 'puppet1.mydomain.com' {
  include motd
  include ntp

  class { 'puppet' :
    puppet_server    => 'puppet.mydomain.com',
    version          => 'latest',
    master           => true,
    agent            => true,
    autosign         => '$confdir/autosign.conf',
    manifest         => '$confdir/environments/$environment/manifests/site.pp',
    modulepath       => '$confdir/environments/$environment/modules',
    puppet_passenger => true,
  }
}

The specific module and init.pp are at https://github.com/olindata/puppetlabs-puppet/blob/master/manifests/init.pp. Line 181 contains the closing } for the 'class { puppet::master:` statement.

I can see that the init.pp does pass puppet_server to the puppet::agent class:

  if $agent {
    class {'puppet::agent':
      version                   => $version,
      puppet_defaults           => $puppet_defaults,
      puppet_agent_service      => $puppet_agent_service,
      puppet_server             => $puppet_server,
      environment               => $environment,
      puppet_conf               => $puppet_conf,
      puppet_agent_name         => $puppet_agent_name,
      package_provider          => $package_provider,
      reporting                 => $reporting,
      user_id                   => $user_id,
      group_id                  => $group_id,  
    }
  }

WHat could be wrong? Am I missing something obvious here?

I am receiving the following error when trying to use the latest puppetlabs-apache module from github (https://github.com/puppetlabs/puppetlabs-apache):

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter identifier at /etc/puppet/environments/development/modules/apache/manifests/mod.pp:40 on node cacti.mydomain.com Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping runn

My node config looks like:

node 'cacti.eye.fi' inherits 'base' {
  include apache
  include mysql::server
  include yumrepos::epel

  package { 'cacti':
    ensure => 'latest',
  }
}

mod.pp contents:

define apache::mod (
  $package = undef
) {
  $mod = $name
  include apache::params
  #include apache #This creates duplicate resources in rspec-puppet
  $mod_packages = $apache::params::mod_packages
  $mod_package = $mod_packages[$mod] # 2.6 compatibility hack
  if $package {
    $package_REAL = $package
  } elsif "$mod_package" {
    $package_REAL = $mod_package
  }
  $mod_libs = $apache::params::mod_libs
  $mod_lib = $mod_libs[$mod] # 2.6 compatibility hack
  if "${mod_lib}" {
    $lib = $mod_lib
  }

  $mod_identifiers = $apache::params::mod_identifiers
  $mod_identifier = $mod_identifiers[$mod]
  if "${mod_identifier}" {
    $identifier = $mod_identifier
  }

  if $package_REAL {
    package { $package_REAL:
      ensure   => present,
      require  => Package['httpd'],
      before   => A2mod[$mod],
    }
  }

  a2mod { $mod:
    ensure     => present,
    lib        => $lib,
    identifier => $identifier,
    require    => Package['httpd'],
    notify     => Service['httpd'],
  }
}

I have verified that the /var/lib/puppet/lib/puppet/type/a2mod.rb type has the identifier parameter and it is the same MD5 as the server:

Puppet::Type.newtype(:a2mod) do
    @doc = "Manage Apache 2 modules"


ensurable

newparam :name do
   desc "The name of the module to be managed"

   isnamevar

end

newparam :lib do
  desc "The name of the .so library to be loaded"

  defaultto { "mod_#{@resource[:name]}.so" }
end

newparam :identifier do
  desc "Module identifier string used by LoadModule. Default: module-name_module"

  # http://httpd.apache.org/docs/2.2/mod/module-dict.html#ModuleIdentifier

  defaultto { "#{@resource[:name]}_module" }
end

autorequire(:package) { catalog.resource(:package, 'httpd')}


end

I am trying to stop my kickstart CentOS install prompting me for TCP/IP information. After I click through this prompt (keeping IPv4 and IPv6 to their defaults) the installation continues and completes just fine. Below is my kickstart file:

# Andy's super awesome VM kickstart file

install
url --url=http://mirrors.kernel.org/centos/6/os/x86_64
lang en_US.UTF-8
keyboard us
text
%include /tmp/network.ks
rootpw  --iscrypted $6$RA8DyrNTsVJkGIgY$ohZ62HHiOjNnn1yDMZlIu3lQ63D3plGPcbVZtPKE8Oq6Z.IGUgN.kNLkxs/ZymZuluRDWsW2eey5zLOl2G3mp.
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone America/Los_Angeles
bootloader --location=mbr --driveorder=vda --append="crashkernel=auto rhgb quiet"
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
zerombr
clearpart --all --drives=vda --initlabel

part /boot --fstype=ext4 --size=500
part pv.253002 --grow --size=1
volgroup vg1 --pesize=4096 pv.253002
logvol / --fstype=ext4 --name=lv_root --vgname=vg1 --grow --size=1024 --maxsize=51200
logvol swap --name=lv_swap --vgname=vg1 --grow --size=4032 --maxsize=4032

repo --name="CentOS"  --baseurl=http://mirrors.kernel.org/centos/6/os/x86_64 --cost=100
repo --name="Puppet Labs Products"  --baseurl=http://yum.puppetlabs.com/el/6/products/x86_64
repo --name="Puppet Labs Dependencies"  --baseurl=http://yum.puppetlabs.com/el/6/dependencies/x86_64
repo --name="EyeFi"  --baseurl=http://flexo.eye.fi/6/eye-fi-api

%packages
@core
@server-policy
puppet
facter
%end

%pre --erroronfail
#!/bin/bash
for x in `cat /proc/cmdline`; do
        case $x in SERVERNAME*)
                eval $x
                echo "network --onboot yes --device eth0 --bootproto dhcp --hostname ${SERVERNAME}.eye.fi" > /tmp/network.ks
                ;;
        esac;
done
%end

%post
puppet agent --waitforcert 10 --onetime --no-daemon --pluginsync --server puppet.eye.fi
%end

reboot

My kernel arguments are in this following virt-install command that I use to start the install:

virt-install -n zabbix -r 2048 --vcpus=2 -l http://mirrors.kernel.org/centos/6/os/x86_64 --disk /dev/vg_inf1/zabbix --network bridge=br85 --initrd-inject=/home/ashinn/vm_kickstart --extra-args "ks=file:/vm_kickstart SERVERNAME=zabbix" --autostart

During the install, I can pull up a console on the second terminal and verify the contents of /tmp/network.ks are:

network --onboot=yes --bootproto=dhcp --ipv6=auto --hostname=jenkins2.mydomain.com

Why might Anaconda be prompting for the TCP/IP settings when they are already set to DHCP?

My Puppet master is using Passenger to serve. After upgrading to Puppet 3.0.0 I am getting the following error:

[ pid=17576 thr=70231398486460 file=utils.rb:176 time=2012-10-01 17:37:12.892 ]: *** Exception NoMethodError in PhusionPassenger::Rack::ApplicationSpawner (undefined method `-@' for "master":String) (process 17576, thread #):
        from config.ru:7
        from /usr/lib/ruby/gems/1.8/gems/rack-1.4.1/lib/rack/builder.rb:51:in `instance_eval'
        from /usr/lib/ruby/gems/1.8/gems/rack-1.4.1/lib/rack/builder.rb:51:in `initialize'
        from config.ru:1:in `new'
        from config.ru:1

My config.ru is as follows:

# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $LOAD_PATH.unshift('/opt/puppet/lib')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"

# Rack applications typically don't start as root.  Set --confdir to prevent
# reading configuration from ~/.puppet/puppet.conf
ARGV << "--confdir" << "/etc/puppet"

# NOTE: it's unfortunate that we have to use the "CommandLine" class
#  here to launch the app, but it contains some initialization logic
#  (such as triggering the parsing of the config file) that is very
#  important.  We should do something less nasty here when we've
#  gotten our API and settings initialization logic cleaned up.
#
# Also note that the "$0 = master" line up near the top here is
#  the magic that allows the CommandLine class to know that it's
#  supposed to be running master.
#
# --cprice 2012-05-22

require 'puppet/util/command_line'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Util::CommandLine.new.execute

Any idea what may be happening?

I have 2 routers announcing a /22 subnet to different providers (one providers connects to each of the 2 routers). I have split the /22 in two /23 to announce one /23 on each of the routers plus the /22 (the providers will take the more specific route). This allows me to fail over and keep traffic inside the /23 in and out the same provider.

What are other ways in which I could announce just the /22 with both routers and have packets from servers on the network behind the routers go back out the same router in which they came in from?

EDIT:

The main problem I come across, which end users and clients complain about the most, is that the least hop route is sometimes not the "optimal" route. In my case, I know that Provider B may have better latency to X nation. But when packets come in from provider B, they may go out Provider A or provider B. The reverse is also true. If I send a packet to X nation out provider A, even though it may have more hops back, the packet will likely come in from Provider B (which may have higher latency, packet loss, etc. to this nation)

I am having problems getting my domain name set correctly on CentOS 6. My resolv.conf looks like this:

nameserver 198.6.100.25
nameserver 198.6.100.53
nameserver 198.6.100.38
# Generated by NetworkManager
domain andyshinn.as
search vcolo.andyshinn.as

I have a single eth0 interface with the following configuration:

DEVICE=eth0
BOOTPROTO=none
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
HWADDR=00:0c:29:3b:2a:70
IPADDR=192.168.50.6
NETMASK=255.255.255.0
DNS2=198.6.100.53
GATEWAY=192.168.50.1
DNS1=198.6.100.25
IPV6INIT=no
USERCTL=no

Hosts file for reference also:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.50.6    myvm myvm.vcolo.andyshinn.as

I am trying to change the domain to vcolo.andyshinn.as. But after reboot, it reverts to andyshinn.as. By the inline comment, I can see it is generated by NetworkManager. However, system-config-network-tui doesn't have anything about the domain name (only the search path).

A fgrep -r 'andyshinn.as' /etc/* only brings up /etc/sysconfig/networking/profiles/default/resolv.conf as another choice to edit. But I have tried editing this file also with the same results. This is a virtual machine running in VMware Workstation. But I have the DHCP server disabled for the VMnet interface that the VM is using.

Rather than just blindly disable NetworkManager (which seems to be the answer for most NetworkManager related issues), I would like to understand more about how it works and saves configuration. So how is my system ending up with andyshinn.as as the domain name and what needs to be updated to set it to vcolo.andyshinn.as?

I am wondering if there are any Linux tools out there which might aid in identification of hosts directly connected to each other (much like Cisco Discovery Protocol on switches)?

I have found http://sourceforge.net/projects/lcdpd/ which searching. But it appears to be a dead project and the patch is only for older kernels.

I am just trying to alleviate the tedious task of ethernet interface to ARP MAC address to hostname lookup. A bonus would be something that also announced its own neighbors so you could have a 2 hop view of the rest of network.

I have 2 Foundry FESX448 switches which have external BGP connections. Each of them receive a default route. These 2 switches also have a BGP session configured between themselves and advertise routes to each other as iBGP.

I only want one of the switches to be the primary gateway as I am trying to move the traffic off the other switch. I tried limiting the announcement of the default gateway using the following prefix list configured as the outbound filter for the primary switch:

prefix-list deny_default: 1 entries
     seq 5 deny 0.0.0.0/0

But this seems to block the default gateway and other subnets connected on the switch. I am only trying to block the default gateway. But I still need to get to directly connected and static routes on the secondary switch.

already have default originate removed from the neighbor BGP config. The received default route from my ISP is the one being re-advertised.

What is a good way to accomplish this?

I am working with 2 x Supermicro SC837E26-RJBOD1 chassis connected to a single LSI 9285-8e card in a Supermicro 1U host. There are 28 drives in each chassis for a total of 56 drives in 28 RAID1 mirrors.

The problem I am running in to is that there are duplicate slots for the 2 chassis (the slots list twice and only go from 0 to 27). All the drives also show the same enclosure ID (ID 36). However, MegaCLI -encinfo lists the 2 enclosures correctly (ID 36 and ID 65).

My question is, why would this happen? Is there an option I am missing to use 2 enclosures effectively?

This is blocking me rebuilding a drive that failed in slot 11 since I can only specify enclosure and slot as parameters to replace a drive. When I do this, it picks the wrong slot 11 (device ID 46 instead of device ID 19).

Adapter #1 is the LSI 9285-8e, adapter #0 (which I removed due to space limitations) is the onboard LSI.

Adapter information:

Adapter #1

==============================================================================
                    Versions
                ================
Product Name    : LSI MegaRAID SAS 9285-8e
Serial No       : SV12704804
FW Package Build: 23.1.1-0004

                    Mfg. Data
                ================
Mfg. Date       : 06/30/11
Rework Date     : 00/00/00
Revision No     : 00A
Battery FRU     : N/A

                Image Versions in Flash:
                ================
BIOS Version       : 5.25.00_4.11.05.00_0x05040000
WebBIOS Version    : 6.1-20-e_20-Rel
Preboot CLI Version: 05.01-04:#%00001
FW Version         : 3.140.15-1320
NVDATA Version     : 2.1106.03-0051
Boot Block Version : 2.04.00.00-0001
BOOT Version       : 06.253.57.219

                Pending Images in Flash
                ================
None

                PCI Info
                ================
Vendor Id       : 1000
Device Id       : 005b
SubVendorId     : 1000
SubDeviceId     : 9285

Host Interface  : PCIE

ChipRevision    : B0

Number of Frontend Port: 0
Device Interface  : PCIE

Number of Backend Port: 8
Port  :  Address
0        5003048000ee8e7f
1        5003048000ee8a7f
2        0000000000000000
3        0000000000000000
4        0000000000000000
5        0000000000000000
6        0000000000000000
7        0000000000000000

                HW Configuration
                ================
SAS Address      : 500605b0038f9210
BBU              : Present
Alarm            : Present
NVRAM            : Present
Serial Debugger  : Present
Memory           : Present
Flash            : Present
Memory Size      : 1024MB
TPM              : Absent
On board Expander: Absent
Upgrade Key      : Absent
Temperature sensor for ROC    : Present
Temperature sensor for controller    : Absent

ROC temperature : 70  degree Celcius

                Settings
                ================
Current Time                     : 18:24:36 3/13, 2012
Predictive Fail Poll Interval    : 300sec
Interrupt Throttle Active Count  : 16
Interrupt Throttle Completion    : 50us
Rebuild Rate                     : 30%
PR Rate                          : 30%
BGI Rate                         : 30%
Check Consistency Rate           : 30%
Reconstruction Rate              : 30%
Cache Flush Interval             : 4s
Max Drives to Spinup at One Time : 2
Delay Among Spinup Groups        : 12s
Physical Drive Coercion Mode     : Disabled
Cluster Mode                     : Disabled
Alarm                            : Enabled
Auto Rebuild                     : Enabled
Battery Warning                  : Enabled
Ecc Bucket Size                  : 15
Ecc Bucket Leak Rate             : 1440 Minutes
Restore HotSpare on Insertion    : Disabled
Expose Enclosure Devices         : Enabled
Maintain PD Fail History         : Enabled
Host Request Reordering          : Enabled
Auto Detect BackPlane Enabled    : SGPIO/i2c SEP
Load Balance Mode                : Auto
Use FDE Only                     : No
Security Key Assigned            : No
Security Key Failed              : No
Security Key Not Backedup        : No
Default LD PowerSave Policy      : Controller Defined
Maximum number of direct attached drives to spin up in 1 min : 10
Any Offline VD Cache Preserved   : No
Allow Boot with Preserved Cache  : No
Disable Online Controller Reset  : No
PFK in NVRAM                     : No
Use disk activity for locate     : No

                Capabilities
                ================
RAID Level Supported             : RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
Supported Drives                 : SAS, SATA

Allowed Mixing:

Mix in Enclosure Allowed
Mix of SAS/SATA of HDD type in VD Allowed

                Status
                ================
ECC Bucket Count                 : 0

                Limitations
                ================
Max Arms Per VD          : 32
Max Spans Per VD         : 8
Max Arrays               : 128
Max Number of VDs        : 64
Max Parallel Commands    : 1008
Max SGE Count            : 60
Max Data Transfer Size   : 8192 sectors
Max Strips PerIO         : 42
Max LD per array         : 16
Min Strip Size           : 8 KB
Max Strip Size           : 1.0 MB
Max Configurable CacheCade Size: 0 GB
Current Size of CacheCade      : 0 GB
Current Size of FW Cache       : 887 MB

                Device Present
                ================
Virtual Drives    : 28
  Degraded        : 0
  Offline         : 0
Physical Devices  : 59
  Disks           : 56
  Critical Disks  : 0
  Failed Disks    : 0

                Supported Adapter Operations
                ================
Rebuild Rate                    : Yes
CC Rate                         : Yes
BGI Rate                        : Yes
Reconstruct Rate                : Yes
Patrol Read Rate                : Yes
Alarm Control                   : Yes
Cluster Support                 : No
BBU                             : No
Spanning                        : Yes
Dedicated Hot Spare             : Yes
Revertible Hot Spares           : Yes
Foreign Config Import           : Yes
Self Diagnostic                 : Yes
Allow Mixed Redundancy on Array : No
Global Hot Spares               : Yes
Deny SCSI Passthrough           : No
Deny SMP Passthrough            : No
Deny STP Passthrough            : No
Support Security                : No
Snapshot Enabled                : No
Support the OCE without adding drives : Yes
Support PFK                     : Yes
Support PI                      : No
Support Boot Time PFK Change    : Yes
Disable Online PFK Change       : No
PFK TrailTime Remaining         : 0 days 0 hours
Support Shield State            : Yes
Block SSD Write Disk Cache Change: Yes

                Supported VD Operations
                ================
Read Policy          : Yes
Write Policy         : Yes
IO Policy            : Yes
Access Policy        : Yes
Disk Cache Policy    : Yes
Reconstruction       : Yes
Deny Locate          : No
Deny CC              : No
Allow Ctrl Encryption: No
Enable LDBBM         : No
Support Breakmirror  : No
Power Savings        : Yes

                Supported PD Operations
                ================
Force Online                            : Yes
Force Offline                           : Yes
Force Rebuild                           : Yes
Deny Force Failed                       : No
Deny Force Good/Bad                     : No
Deny Missing Replace                    : No
Deny Clear                              : No
Deny Locate                             : No
Support Temperature                     : Yes
Disable Copyback                        : No
Enable JBOD                             : No
Enable Copyback on SMART                : No
Enable Copyback to SSD on SMART Error   : Yes
Enable SSD Patrol Read                  : No
PR Correct Unconfigured Areas           : Yes
Enable Spin Down of UnConfigured Drives : Yes
Disable Spin Down of hot spares         : No
Spin Down time                          : 30
T10 Power State                         : Yes
                Error Counters
                ================
Memory Correctable Errors   : 0
Memory Uncorrectable Errors : 0

                Cluster Information
                ================
Cluster Permitted     : No
Cluster Active        : No

                Default Settings
                ================
Phy Polarity                     : 0
Phy PolaritySplit                : 0
Background Rate                  : 30
Strip Size                       : 64kB
Flush Time                       : 4 seconds
Write Policy                     : WB
Read Policy                      : Adaptive
Cache When BBU Bad               : Disabled
Cached IO                        : No
SMART Mode                       : Mode 6
Alarm Disable                    : Yes
Coercion Mode                    : None
ZCR Config                       : Unknown
Dirty LED Shows Drive Activity   : No
BIOS Continue on Error           : No
Spin Down Mode                   : None
Allowed Device Type              : SAS/SATA Mix
Allow Mix in Enclosure           : Yes
Allow HDD SAS/SATA Mix in VD     : Yes
Allow SSD SAS/SATA Mix in VD     : No
Allow HDD/SSD Mix in VD          : No
Allow SATA in Cluster            : No
Max Chained Enclosures           : 16
Disable Ctrl-R                   : Yes
Enable Web BIOS                  : Yes
Direct PD Mapping                : No
BIOS Enumerate VDs               : Yes
Restore Hot Spare on Insertion   : No
Expose Enclosure Devices         : Yes
Maintain PD Fail History         : Yes
Disable Puncturing               : No
Zero Based Enclosure Enumeration : No
PreBoot CLI Enabled              : Yes
LED Show Drive Activity          : Yes
Cluster Disable                  : Yes
SAS Disable                      : No
Auto Detect BackPlane Enable     : SGPIO/i2c SEP
Use FDE Only                     : No
Enable Led Header                : No
Delay during POST                : 0
EnableCrashDump                  : No
Disable Online Controller Reset  : No
EnableLDBBM                      : No
Un-Certified Hard Disk Drives    : Allow
Treat Single span R1E as R10     : No
Max LD per array                 : 16
Power Saving option              : Don't Auto spin down Configured Drives
Max power savings option is  not allowed for LDs. Only T10 power conditions are to be used.
Default spin down time in minutes: 30
Enable JBOD                      : No
TTY Log In Flash                 : No
Auto Enhanced Import             : No
BreakMirror RAID Support         : No
Disable Join Mirror              : No
Enable Shield State              : Yes
Time taken to detect CME         : 60s

Exit Code: 0x00

Enclosure information:

# /opt/MegaRAID/MegaCli/MegaCli64 -encinfo -a1

    Number of enclosures on adapter 1 -- 3

    Enclosure 0:
    Device ID                     : 36
    Number of Slots               : 28
    Number of Power Supplies      : 2
    Number of Fans                : 3
    Number of Temperature Sensors : 1
    Number of Alarms              : 1
    Number of SIM Modules         : 0
    Number of Physical Drives     : 28
    Status                        : Normal
    Position                      : 1
    Connector Name                : Port B
    Enclosure type                : SES
    VendorId is LSI CORP and Product Id is SAS2X36
VendorID and Product ID didnt match
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : 65

    Inquiry data                  :
        Vendor Identification     : LSI CORP
        Product Identification    : SAS2X36
        Product Revision Level    : 0718
        Vendor Specific           : x36-55.7.24.1

Number of Voltage Sensors         :2

Voltage Sensor                    :0
Voltage Sensor Status             :OK
Voltage Value                     :5020 milli volts

Voltage Sensor                    :1
Voltage Sensor Status             :OK
Voltage Value                     :11820 milli volts

Number of Power Supplies     : 2

Power Supply                 : 0
Power Supply Status          : OK

Power Supply                 : 1
Power Supply Status          : OK

Number of Fans               : 3

Fan                          : 0
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 1
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 2
Fan Speed              :Low Speed
Fan Status                   : OK

Number of Temperature Sensors : 1

Temp Sensor                  : 0
Temperature                  : 48
Temperature Sensor Status    : OK

Number of Chassis             : 1

Chassis                      : 0
Chassis Status               : OK

    Enclosure 1:
    Device ID                     : 65
    Number of Slots               : 28
    Number of Power Supplies      : 2
    Number of Fans                : 3
    Number of Temperature Sensors : 1
    Number of Alarms              : 1
    Number of SIM Modules         : 0
    Number of Physical Drives     : 28
    Status                        : Normal
    Position                      : 1
    Connector Name                : Port A
    Enclosure type                : SES
    VendorId is LSI CORP and Product Id is SAS2X36
VendorID and Product ID didnt match
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : 36

    Inquiry data                  :
        Vendor Identification     : LSI CORP
        Product Identification    : SAS2X36
        Product Revision Level    : 0718
        Vendor Specific           : x36-55.7.24.1

Number of Voltage Sensors         :2

Voltage Sensor                    :0
Voltage Sensor Status             :OK
Voltage Value                     :5020 milli volts

Voltage Sensor                    :1
Voltage Sensor Status             :OK
Voltage Value                     :11760 milli volts

Number of Power Supplies     : 2

Power Supply                 : 0
Power Supply Status          : OK

Power Supply                 : 1
Power Supply Status          : OK

Number of Fans               : 3

Fan                          : 0
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 1
Fan Speed              :Low Speed
Fan Status                   : OK

Fan                          : 2
Fan Speed              :Low Speed
Fan Status                   : OK

Number of Temperature Sensors : 1

Temp Sensor                  : 0
Temperature                  : 47
Temperature Sensor Status    : OK

Number of Chassis             : 1

Chassis                      : 0
Chassis Status               : OK

    Enclosure 2:
    Device ID                     : 252
    Number of Slots               : 8
    Number of Power Supplies      : 0
    Number of Fans                : 0
    Number of Temperature Sensors : 0
    Number of Alarms              : 0
    Number of SIM Modules         : 1
    Number of Physical Drives     : 0
    Status                        : Normal
    Position                      : 1
    Connector Name                : Unavailable
    Enclosure type                : SGPIO
Failed in first Inquiry commnad
    FRU Part Number               : N/A
    Enclosure Serial Number       : N/A
    ESM Serial Number             : N/A
    Enclosure Zoning Mode         : N/A
    Partner Device Id             : Unavailable

    Inquiry data                  :
        Vendor Identification     : LSI
        Product Identification    : SGPIO
        Product Revision Level    : N/A
        Vendor Specific           :


Exit Code: 0x00

Now, notice that each slot 11 device shows an enclosure ID of 36, I think this is where the discrepancy happens. One should be 36. But the other should be on enclosure 65.

Drives in slot 11:

Enclosure Device ID: 36
Slot Number: 11
Drive's postion: DiskGroup: 5, Span: 0, Arm: 1
Enclosure position: 0
Device Id: 48
WWN:
Sequence Number: 11
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 2.728 TB [0x15d50a3b0 Sectors]
Non Coerced Size: 2.728 TB [0x15d40a3b0 Sectors]
Coerced Size: 2.728 TB [0x15d400000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : YES
Device Firmware Level: A5C0
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5003048000ee8a53
Connected Port Number: 1(path0)
Inquiry Data:       MJ1311YNG6YYXAHitachi HDS5C3030ALA630                 MEAOA5C0
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :30C (86.00 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's write cache : Disabled
Drive's NCQ setting : Enabled
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No



Enclosure Device ID: 36
Slot Number: 11
Drive's postion: DiskGroup: 19, Span: 0, Arm: 1
Enclosure position: 0
Device Id: 19
WWN:
Sequence Number: 4
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 2.728 TB [0x15d50a3b0 Sectors]
Non Coerced Size: 2.728 TB [0x15d40a3b0 Sectors]
Coerced Size: 2.728 TB [0x15d400000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : NO
Device Firmware Level: A580
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5003048000ee8e53
Connected Port Number: 0(path0)
Inquiry Data:       MJ1313YNG1VA5CHitachi HDS5C3030ALA630                 MEAOA580
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :30C (86.00 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Drive's write cache : Disabled
Drive's NCQ setting : Enabled
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

Update 06/28/12:

I finally have some new information about (what we think) the root cause of this problem so I thought I would share.

After getting in contact with a very knowledgeable Supermicro tech, they provided us with a tool called Xflash (doesn't appear to be readily available on their FTP). When we gathered some information using this utility, my colleague found something very strange:

root@mogile2 test]# ./xflash.dat -i get avail

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


1) SAS2X36 (SAS2x36) (50030480:00EE917F)  (0.0.0.0) 
2) SAS2X36 (SAS2x36) (50030480:00E9D67F)  (0.0.0.0) 
3) SAS2X36 (SAS2x36) (50030480:0112D97F)  (0.0.0.0)

This lists the connected enclosures. You see the 3 connected (we have since added a 3rd and a 4th which is not yet showing up) with their respective SAS address / WWN (50030480:00EE917F). Now we can use this address to get information on the individual enclosures:

[root@mogile2 test]# ./xflash.dat -i 5003048000EE917F get exp 

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


Reading the expander information..........

            Expander: SAS2X36 (SAS2x36) B3
         SAS Address: 50030480:00EE917F
Enclosure Logical Id: 50030480:0000007F
          IP Address: 0.0.0.0
Component Identifier: 0x0223
  Component Revision: 0x05

[root@mogile2 test]# ./xflash.dat -i 5003048000E9D67F get exp                 

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


Reading the expander information..........

            Expander: SAS2X36 (SAS2x36) B3
         SAS Address: 50030480:00E9D67F
Enclosure Logical Id: 50030480:0000007F
          IP Address: 0.0.0.0
Component Identifier: 0x0223
  Component Revision: 0x05

[root@mogile2 test]# ./xflash.dat -i 500304800112D97F get exp                 

Initializing Interface.
Expander: SAS2X36 (SAS2x36)


Reading the expander information..........

            Expander: SAS2X36 (SAS2x36) B3
         SAS Address: 50030480:0112D97F
Enclosure Logical Id: 50030480:0112D97F
          IP Address: 0.0.0.0
Component Identifier: 0x0223
  Component Revision: 0x05

Did you catch it? The first 2 enclosures logical ID is partially masked out where the 3rd one (which has a correct unique enclosure ID) is not. We pointed this out to Supermicro and were able to confirm that this address is supposed to be set during manufacturing and there was a problem with a certain batch of these enclosures where the logical ID was not set.

We believe that the RAID controller is determining the ID based on the logical ID and since our first 2 enclosures have the same logical ID, they get the same enclosure ID. We also confirmed that 0000007F is the default which comes from LSI as an ID.

The next pointer that helps confirm this could be a manufacturing problem with a run of JBODs is the fact that all 6 of the enclosures that have this problem begin with 00E. I believe that between 00E8 and 00EE Supermicro forgot to program the logical IDs correctly and neglected to recall or fix the problem post production.

Fortunately for us, there is a tool to manage the WWN and logical ID of the devices from Supermicro: ftp://ftp.supermicro.com/utility/ExpanderXtools_Lite/. Our next step is to schedule a shutdown of these JBODs (after data migration) and reprogram the logical ID and see if it solves the problem.

Update 06/28/12 #2:

I just discovered this FAQ at Supermicro while Google searching for "lsi 0000007f": http://www.supermicro.com/support/faqs/faq.cfm?faq=11805. I still don't understand why, in the last several times we contacted Supermicro, they would have never directed us to this article :\

I am searching for information on mapping an existing drive (USB drive) to a KVM virtual machine using libvirt. I have been going over http://libvirt.org/formatstorage.html#StoragePoolTarget. But I don't quite understand what options are needed to expose an existing device to the VM (or if it is even possible).

From what I am reading, the target element is expecting a path element to a folder on the machine to store a file based image. Can the path be a device such as /dev/sdc1? with a type of raw?

I am testing out a iSCSI SAN solution in my vSphere environment and have been adding/removing LUNs on the SAN host while some were left connected on ESXi hosts. I had enabled Storage I/O Control on the datastores and now I cannot remove them. The errors received is:

Call "HostDatastoreSystem.RemoveDatastore" for object "datastoreSystem-580" on vCenter Server "VCENTER1" failed.
Cannot remove datastore 'ramdisk' because Storage I/O Control is enabled on it. Correct it and re-try the operation

I cannot disable Storage I/O Control since I cannot right click on the datastore and get to the Properties option. There is a KB article for removing inaccessible NFS datastores:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2008507

But this doesn't seem to apply to iSCSI datastores. I no longer have the LUN available to represent.

I have also tried unlclaiming the storage device and get an error that it is still busy:

~ # esxcli corestorage claiming unclaim -t device -d naa.600144f0a4a4450000004d7ae2dd0002
Errors:
Unable to perform unclaim.  Error message was : Unable to unclaim all requested paths. Some paths were busy or were the last path to an in use device.  See VMkernel logs for more information.

How can I remove the iSCSI datastore that is missing the LUN?

I am trying to add a calculated item based on a web scenario response time. I am trying to add an item like so:

avg("web.test.time[fut_perf,index,resp]", #5)+avg("web.test.time[fut_perf,about,resp]" #5)

I am hoping to get the combined value of both averages over 5 polls. I am getting the error:

ERROR: Page received incorrect data

    Warning. Incorrect value for [Formula]

I have also tried:

avg("web.test.time[fut_perf,index,resp]", 120)+avg("web.test.time[fut_perf,about,resp]" 120)
avg("web.test.time[fut_perf,index,resp]", 120)
avg(web.test.time[fut_perf,index,resp], 120)+avg(web.test.time[fut_perf,about,resp] 120)
avg(web.test.time[fut_perf,index,resp], 120)
avg("web.test.time[fut_perf,index,resp]")

Zabbix version is 1.8.4. Any ideas what I might be doing wrong? Is it possible that web scenario items are not supported in calculated items?