EEAA's questions

I have been using Ansible, to great success, for ~3 years now for managing an ever-growing flock of linux systems. Before diving into my question, I need to set some context.

As part of my day job, I do systems design, deployment, and maintenance for various companies that all operate under the umbrella of a single venture/incubator company. There is a lot of cross-pollination among our portfolio companies, and as such, we cannot say that only users A, B, and C will need access to company X's systems. They may also need access to company Y's systems as well. This is complicated by the fact that each company's ansible environment lives in a different git repository. This means that there is a lot of code duplication for deploying users to different company's systems. I end up copy/pasting blocks of code like this around to deploy users to a certain company's systems:

- name: add several users
  user: >
    name={{ item.name }}
    state=present
    groups={{ item.groups }}
    uid={{ item.uid }}
    password={{ item.password }}
    shell=/bin/bash
  with_items:
    - { name: 'user1', groups: 'ssh-access,sudo', uid: '501', password: '<redacted>' }
    - { name: 'user2', groups: 'ssh-access,sudo', uid: '502', password: '<redacted>' }
  tags: users

- name: authorized_keys - user1 
  action: authorized_key user=user1 key="{{ lookup('file', 'pubkeys/user1') }}" manage_dir=yes
  tags:
    - pubkeys
    - users

- name: authorized_keys - user2 
  action: authorized_key user=user2 key="{{ lookup('file', 'pubkeys/user2') }}" manage_dir=yes
  tags:
    - pubkeys
    - users

This worked OK when I had a <5 users to manage, but as the user base grows, it becomes more and more onerous to keep things up to date with key rotation, new passwords, etc.

With the backstory and context set, on with my question:

Assuming that using a centralized authentication system (LDAP, etc.) is not an option, how could I tackle creating a centralized user database that various ansible playbooks could consume? I would love to be able to maintain one central list of users, uids, password hashes, and public keys, and then be able to deploy the users (with custom per-host group memberships) to each company's hosts.

I'm envisioning some sort of play structure like:

- name: Deploy users
  user_management:
    - { name: "user1", groups: "sudo" }
    - { name: "user1", groups: "sudo" }

...where each user's uid, hash, and public key would be pulled from the central list and deployed as usual.

So, what options do I have? I've been mulling this around for quite a while, and haven't been able to come up with anything better than what I'm already doing. Could I do something with a custom facts file to hold my user database?

I have a cfn stack that (among other things), creates a VPC, several security groups, and a handful of EC2 instances. It's trivial to assign security groups that are created within the stack to instances that are also created by the stack. However, I'm interested in the default VPC SG.

When a VPC gets created (whether manually though the GUI, by cloudformation, or any other means), AWS creates a default security group with an "allow all" rule for any instance in that group.

What I am trying to do is assign this default security group along with several other SGs to instances created by the stack. This is proving to be far more difficult than I anticipated. Here are some snippets showing what I have going on:

"AllowSSHSecGroup":{
      "Type":"AWS::EC2::SecurityGroup",
      "Properties":{
        "GroupDescription":"Allow SSH from anywhere",
        "VpcId":{
          "Ref":"DevVPC"
        },
        "SecurityGroupIngress":[
          {
            "IpProtocol":"tcp",
            "FromPort":"22",
            "ToPort":"22",
            "CidrIp":"0.0.0.0/0"
          }
        ]
      }
},
"Instance001" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : "ami-7eab224e",
        "InstanceType" : "m1.large",
        "AvailabilityZone" : "us-west-2a",
        "PrivateIpAddress" : "10.22.0.110",
        "SecurityGroupIds" : [ {"Ref" : "AllowSSHSecGroup"} ],
        "SubnetId" : { "Ref" : "PublicSubnet" },
        "KeyName" : "erik-key",
        "DisableApiTermination" : "false",
        "Tags" : [ { "Key": "Name", "Value": "Instance001"} ]
      }
}

In the above snippet, I'm creating an "allow ssh" security group and assigning that to an instance. As mentioned, my stack also creates a VPC (which this instance is launched in), which in turn creates a default security group. Unfortunately, since this group is created automatically by AWS, its group ID is unavailable to the stack, making it impossible to reference by ID. I initially thought that the SecurityGroups property would be an option, as that would allow me to reference the default SG by its name, default. That doesn't work, though, as the SecurityGroups property is only for EC2 Security Groups, not VPC Security Groups.

So I'm stuck. I have opened up a case with AWS support on this, but so far, they've not been helpful. Any ideas on how I can accomplish this?

Yesterday afternoon, we resized one of our Linode instances (CentOS 5.7, 64-bit) from a 4GB instance to 12GB. Immediately following that reboot, I noticed that memory usage for buffers was insanely high - higher than I've ever seen on any machine I've touched. Even on my most heavily-used servers, I rarely see buffer usage exceed ~200MB. On this server, the current buffer usage is two orders of magnitude higher than before we resized and rebooted.

Here is the a munin memory graph with data pre and post migration:

Data that munin is displaying is corroborated by the output of "free":

[erik@host ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:         11967      10146       1820          0       7374       1132
-/+ buffers/cache:       1639      10327
Swap:          255          0        255

Now, I'm well aware of the kernel's usage of unused memory for cache, but my understanding of buffers is that buffers are different. They're used to temporarily store writes until they've been committed to disk. Is that a correct understanding? This server has very little disk IO (it's an apache/php webserver, DB is elsewhere, so only IO of substance are access_logs), and as such, I'd expect buffer usage to be quite low.

Here is a network traffic graph for the same time period:

As you can see, there is no substantive change in traffic before and after the resize.

During the reboot, three things changed that I know of:

1. We picked up 4 additional cores that Linode gave out earlier this week, bring the total cores to 8.
2. We're on the "Latest 64 bit" kernel, which is now 3.7.10-x86_64-linode30. Previously we were on 3.0.18 I believe.
3. We went from 4GB RAM to 12GB.

Of these changes, my hunch is that it was the new kernel that is causing the increased buffer usage. Unfortunately, at the moment, we can't take another downtime hit to downgrade to an earlier kernel, though that may end up being necessary if I can't get this buffer usage sorted out.

With that, I have a couple questions:

1. Are any of you running the 3.7.10 kernel and if so, have you seen a similar change?
2. What tools are available to inspect the kernel buffers and their sizes?
3. I assume that, like cache, the kernel will release this memory when other applications need it. Is this correct?

We're building a medical image processing software stack, currently hosted on various AWS resources. As part of this application, we have a handful of long-running servers (database, load balancers, web application, etc.). Collecting performance data on those servers is quite simple - my go-to- recipe of Nagios (for monitoring/notifications) and Munin (for collection of performance data and displaying trends) will work just fine.

However - as part of this application, we are constantly starting up and terminating compute instances on EC2. In typical usage, these compute instances start up, configure themselves, receive a job from a message queue, and then get to work processing that job, which takes anywhere from 15 minutes to over 8 hours. After job completion, these instances get terminated, never to be heard from again.

What is a decent strategy for collecting performance data on these short-lived instances?

I don't necessarily need monitoring on them - if they fail for whatever reason, our application will detect this and handle re-starting the job on another instance or raising the flag so an administrator can take a look at things. However, it still would be useful to collect information like CPU (user, idle, iowait, etc.), memory usage, network traffic, disk read/write data, etc. In our internal database, we track the instance ID of the machine that runs each job, and it would be quite helpful to be able to look up performance data for a specific instance ID for troubleshooting and profiling.

Munin doesn't seem like a great candidate, as it requires maintaining a list of munin nodes in a text file - far from ideal for an environment with a high amount of churn, and for the short amount of time each node will be running, I'd rather keep the full-resolution data indefinitely than have RRD water down the data over time.

In the end, my guess is that this will require a monitoring engine that:

• uses a database (MySQL, SQLite, etc.) for configuration and data storage
• exposes an API for adding/removing hosts and services

Are there other things I should be thinking about when evaluating options?

Perhaps I'm over-thinking this, though, and just ought to run sar at 1-minute intervals on these short-lived instances and collect the sar db files prior to termination.

Background

We're using nagios to monitor our infrastructure. We don't have the nagios configs under version control at the moment, and there are two of us that manage nagios configuration. As such, I'm working to get our nagios config into a central git repo, using some hooks to do syntax checking and then if the configs look good, make them "active". I'm using this guy's post as a starting point.

The general workflow I'm trying to implement is:

1. Edit local git repo of nagios config. Add edited files, commit locally.
2. git push origin master to the remote repo.
3. Push is intercepted by the pre-receive hook, which takes the files, moves them to a temporary directory on the server, and runs them through the nagios syntax checker.
4. If the syntax checker passes, accept the push, then use the post-commit hook to git pull the new code into the live nagios configuration directory and then restart nagios.
5. If the syntax checker fails, reject the push, showing the nagios syntax error to the user.

I'm running into an odd behavior, though, when I reject a git push due to syntax errors in the nagios config. What I expect to happen is that if I reject the hook, the attempted push should leave the repository just how it was, untouched. That doesn't appear to be the case, though. Below are the details of what I'm seeing:

Problem

I edit the nagios config locally, intentionally including a syntax error, add, then commit locally:

host:nagios erik$ vi nagios.cfg
host:nagios erik$ git add nagios.cfg
host:nagios erik$ git commit -m "syntax error"
[master da71aed] syntax error
 1 files changed, 1 insertions(+), 0 deletions(-)

Now I push those changes to the master repo. This will be rejected due to the syntax error:

host:nagios erik$ git push origin master
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 12.74 KiB, done.
Total 3 (delta 1), reused 2 (delta 1)
remote: Previous HEAD position was 3ddc880... removed syntax error
remote: HEAD is now at da71aed... syntax error
remote: Nagios Config Check Exit Status: 254
remote: Your configs did not parse correctly, there was an error. Output follows.
remote:
remote: Nagios Core 3.2.3
remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors
remote: Copyright (c) 1999-2009 Ethan Galstad
remote: Last Modified: 10-03-2010
remote: License: GPL
remote:
remote: Website: http://www.nagios.org
remote: Reading configuration data...
remote: Error in configuration file '/tmp/nagiosworkdir/nagios.cfg' - Line 23 (NULL value)
remote:    Error processing main config file!
remote:
remote:
remote:
remote: ***> One or more problems was encountered while processing the config files...
remote:
remote:      Check your configuration file(s) to ensure that they contain valid
remote:      directives and data defintions.  If you are upgrading from a previous
remote:      version of Nagios, you should be aware that some variables/definitions
remote:      may have been removed or modified in this version.  Make sure to read
remote:      the HTML documentation regarding the config files, as well as the
remote:      'Whats New' section to find out what has changed.
remote:
To [email protected]:nagios
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:nagios'

This shouldn't have touched the remote repo, but it did. If I change to another local temp directory and try to clone the repo, I get:

host:temp erik$ git clone [email protected]:nagios
Cloning into nagios...
remote: Counting objects: 30, done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 30 (delta 12), reused 0 (delta 0)
Receiving objects: 100% (30/30), 29.81 KiB, done.
Resolving deltas: 100% (12/12), done.
error: Trying to write ref HEAD with nonexistant object da71aedfde2e0469288acd9e45bb8b57a6e5a7b3
fatal: Cannot update the ref 'HEAD'.

Now I go back to the original work directory, fix the syntax error, add, commit, and push:

host:nagios erik$ vi nagios.cfg
host:nagios erik$ git add nagios.cfg
host:nagios erik$ git commit -m "removing syntax error, push should succeed this time"
[master f147ded] removing syntax error, push should succeed this time
 1 files changed, 0 insertions(+), 2 deletions(-)
host:nagios erik$ git push origin master
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 487 bytes, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Previous HEAD position was 4c80d45... syntax error
remote: HEAD is now at f147ded... removing syntax error, push should succeed this time
remote: Nagios Config Check Exit Status: 0
remote: Your configs look good and parsed correctly.
To [email protected]:nagios
   3ddc880..f147ded  master -> master

At this point, the repository is fine, and I'm able to change to a temporary directory and clone the repo again:

host:temp erik$ git clone [email protected]:nagios
Cloning into nagios...
remote: Counting objects: 34, done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 34 (delta 14), reused 0 (delta 0)
Receiving objects: 100% (34/34), 30.22 KiB, done.
Resolving deltas: 100% (14/14), done.

Here is the pre-receive hook I'm using.

I'm using git v1.7.5.4 on the client, and v1.7.2.3 on the server.

So, to the question: why is the repository being left in an inconsistent state when I reject the push? Is something awry with my git hook or perhaps my understanding of git is lacking?

I have several users that use SmartCVS on OSX to interact with our CVS repository. We use the cvs :ext: (SSH2) method to access the repository, which is stored on a remote linux system. We use ssh key authentication, and require that users encrypt their private keys with a passphrase.

The version of ssh-keygen that ships with OSX uses AES-128-CBC to encrypt private keys, and apparently whatever ssh library SmartCVS uses doesn't support decrypting this cipher, as it throws an error. If I generate a key on Windows using PuTTYgen and move that key over to OSX, things work fine. PuTTYgen (and most versions of ssh-keygen on linux) use the DES-EDE3-CBC cipher, which SmartCVS is able to decrypt without issue.

So, my question is: is there a way to tell ssh-keygen which cipher to use when encrypting the private key? The ssh-keygen(1) man page on OSX doesn't say anything about being able to set the cipher type, but I'm hoping there may be some other way to convert to another cipher type after generation.

We recently migrated our entire VMware cluster from ESX over to ESXi. For the most part, the transition was seamless, and I haven't missed having access to the SC. Until now.

We're trying to diagnose some odd unicast flooding behavior that's happening during vMotions, and we suspect that the cause may be related to a discrepancy between switchgear CAM table cache expiry and the ARP table expiry on each ESXi host. As such, I've been trying to figure out how to view and clear the ARP table in ESXi.

On ESX (with the full SC), this would have been a cinch - just ssh in and run an arp -a. Unfortunately the neutered shell within ESXi doesn't include the ARP command, and I have not been able to find a single piece of documentation on this within VMware's KB.

I do have a support request in with VMware on this (going on 30 hours without an answer), but figured I'd toss it over here first to see if anyone has ideas. Thanks!