Ernest Mueller's questions

I have two Cloudformation files being used to make two different stacks containing a VPC each. One is an admin VPC that will be used to access the other VPC via ssh and all that, typical bastiony use case.

I spin up the admin VPC and then pass its ID into the second CF file as a parameter specifically with the VPC Id type:

"AdminVPC": {
  "Description": "ID of the admin VPC",
  "Type": "AWS::EC2::VPC::Id"
}

But then when I am trying to set up the network ACL of the VPC, I am doing

"Type": "AWS::EC2::NetworkAclEntry",
  "Properties": {
    "CidrBlock": {
      "Fn::GetAtt": [
        {
          "Ref": "AdminVPC"
        },
        "CidrBlock"
      ]
    },

Which, when I run a ecs cf validate, yields only the message

An error occurred (ValidationError) when calling the ValidateTemplate operation: Internal Failure

It works fine if I just hardcode in a CIDR block like

"CidrBlock": "0.0.0.0/0",

But the docs claim that:

1. Using a GetAtt to get the CIDR block off the vpc ID should work
2. For the Fn::GetAtt attribute name, you can use the Ref function.

So I'm not sure what is wrong with this usage...

I have a CloudFormation template I'm using to set up an ECS cluster and I am trying to drop some config files onto the box using CloudFormation::Init on the ASG and pulling them out of S3.

"ECSASGLaunchConfiguration": {
  "Type": "AWS::AutoScaling::LaunchConfiguration",
  "Metadata": {
    "AWS::CloudFormation::Authentication": {
      "S3AccessCreds": {
        "type": "S3",
        "roleName": {
          "Ref": "ECSEC2InstanceIAMRole"
        }
      }
    },
    "AWS::CloudFormation::Init": {
      "config": {
        "packages": {
        },
        "groups": {
        },
        "users": {
        },
        "sources": {
        },
        "files": {
          "/etc/dd-agent/conf.d/nginx.yaml": {
            "source": "https://s3.amazonaws.com/foobar/scratch/nginx.yaml",
            "mode": "000644",
            "owner": "root",
            "group": "root"
          },
          "/etc/dd-agent/conf.d/docker_daemon.yaml": {
            "source": "https://s3.amazonaws.com/foobar/scratch/docker_daemon.yaml",
            "mode": "000644",
            "owner": "root",
            "group": "root"
          }
        },
        "commands": {
        },
        "services": {
        }
      }
    }
  },

To do so, I've added a policy inline to the role I'm creating for my EC2 instances that should allow all S3 access from the instances.

"ECSEC2InstanceIAMRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "AssumeRolePolicyDocument": {
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": [
              "ec2.amazonaws.com"
            ]
          },
          "Action": [
            "sts:AssumeRole"
          ]
        }
      ]
    },
    "ManagedPolicyArns": [
      "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
      "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
    ],
    "Path": "/",
    "Policies": [
      {
        "PolicyName": "otxS3access",
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": "s3:ListAllMyBuckets",
              "Resource": "arn:aws:s3:::*"
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
              ],
              "Resource": "arn:aws:s3:::foobar"
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
              ],
              "Resource": "arn:aws:s3:::foobar/*"
            }
          ]
        }
      }
    ]
  }
},
"ECSEC2InstanceIAMProfile": {
  "Type": "AWS::IAM::InstanceProfile",
  "Properties": {
    "Path": "/",
    "Roles": [
      {
        "Ref": "ECSEC2InstanceIAMRole"
      }
    ]
  }
},

But it doesn't work. I can't find an error (or anything else) in the logs. When I manually try to curl it from the instance it doesn't work either, "AccessDenied..."

The bucket itself in this case, "foobar", doesn't have any special permissions besides not being public, just the standard account-name single grantee.

Any idea what I'm doing wrong?

When I try to install gems like ohai and fpm on my Amazon Linux EC2 instance I get the error message:

[ec2-user@ip-172-31-43-16 ~]$ sudo gem install ohai
Building native extensions.  This could take a while...
ERROR:  Error installing ohai:
        ERROR: Failed to build gem native extension.

    /usr/bin/ruby2.0 extconf.rb

Gem files will remain installed in /usr/local/share/ruby/gems/2.0/gems/ffi-1.9.6 for inspection.
Results logged to /usr/local/share/ruby/gems/2.0/gems/ffi-1.9.6/ext/ffi_c/gem_make.out

That's it, no additional hints. The message is identical with fpm.

I've Googled and have done everything that I think I'm supposed to do. I have yum installed gcc, make, install, ruby-devel - in fact in a fit of trying everything I yum installed everything ruby20*, rubygem[s]20*, etc. and yum updated the hell out of my whole system. I have done a gem update --system.

I was able to gem install facter, which doesn't build a native extension, so the rest of Rubyville seems fine.

have a problem I was hoping someone could help me with. We have two RDS read slaves that we have our app servers round-robin to for data reads (DNS in Route53).

Well, yesterday those RDSes got restarted by AWS and changed IPs and it all stopped working. We have a CNAME to the master that the app uses for writes, which is more resilient, but you can't round-robin CNAMEs. You also can't elastic-IP a RDS. What is a good solution to using multiple read slaves from an arbitrary app (not changing the app to make it call the API to figure it out) that's resilient to restarts/IP changes?

I administer a Windows 2008 server (well, on Amazon EC2) running IIS and a .NET4 Web app. I got a memory alert the other day and went and looked, and sure enough the process memory had grown over time via some kind of slow leak. It didn't grow by much, just like 60M to 200M, but enough else was going on with the box that it went over our pretty low threshold (75%) to set off the monitor.

I recycled the app's pool and the memory freed up, and I noticed upon reviewing stats that swap space was being used significantly and that more than 1 GB of it freed up with that recycle.

Maybe this is a basic question, but I'm a UNIX guy and I'm used to swap not getting used until you're out of memory. This box has never gone above 75% memory usage. Is this a Windows thing or a .NET thing or an Amazon thing? I suspect that there's a much larger memory leak in this app than suspected - it's not leaking from 60M to 200M, it's leaking from 60M to 1.2GB, but much of that is somehow going "cold" and being pushed out to swap?

I have memory recycling set on the application pool, but it triggers off box full memory, so this app could get really, really big before it recycles automatically.

I could set up regular "timed" recycling, but that's a workaround, I'll get the dev to fix the app but need to understand what's going on here with the swap usage to make sure I am understanding this right.

Edit with more info: instance memory: 1.7 GB swap: 4.5 GB

I see the w3wp.exe process in taskmgr showing that Memory: 211,000k. But when I restarted it (it's in its own app pool, and it's the only app on the box), its memory usage went down to its normal starting point of 60M and like 1 GB+ of swap also freed up. In taskmgr I just had the usual Memory (Private Working Set) stat up, but saw the swap change via my other monitoring (Cloudkick). Going back and looking at it today, memory is back up to 195M on the process (1.2 GB total) and swap has crept from 1.0 GB to 1.1 GB, buyt not all the way back up where it was (graphing over time, it's a slow creep).

I'm less concerned about this specific app and more concerned about just understanding when Windows swaps and how it uses that, and what to be concerned about given Windows memory and swap usage in general.

I am trying to reproduce a network-related performance problem from a remote site to a cloud-hosted system I'm responsible for. They get awful performance from their location, I get fine performance from mine. I have a hardware WAN emulator (Apposite Linktropy Mini2 in this case) I can use to try to simulate their network to reproduce the problem. But first, I need to know the performance characteristics of the network between them and the server.

What I need is a concise way that I could have an average user run from their PC or whatever that could sum up bandwidth, latency, packet loss, and whatever else I can get. Ideally they'd just put in a DNS name and generate some numbers they can feed me - I can't have them do all kinds of techie work, make them find a Linux box, etc.

What would a good tool be that would characterize the network between two points?

I'm currently backing up/migrating Amazon S3 and SimpleDB data from AWS to my on premise systems using sdbShell and s3cmd. Works great.

Now we're putting an app up on Microsoft Azure and I want to extend my reach to those. Are there any utilities like this for Azure? Ideally they'd work from my existing UNIX host but I could be talked out of that, but they need to be command line and not GUI doodads, I run these nightly and tied to some system automation.

I have a replicated mySQL setup running happily on Amazon AWS, making user data available locally in various regions. Now I'm faced with an app that needs to go up on Microsoft Azure and I need to replicate the data over to there as well.

So that's annoying. I am faced with several options:

1. Replicate from mySQL to SQL Azure/SQL Server seems like it would be lovely - is this possible? I'd consider using a third party tool and paying $$ if I had to. We're not using anything complicated in the db feature set, it's just data in tables.

2. Get mySQL working on Microsoft Azure - which seems really dicey at best. All the HOWTOs I can find say "this is possible but you really shouldn't try this for production apps."

3. Go non-realtime and do syncs from mySQL to SQL Azure, which may be somewhat expensive and slower.

4. Rip out all my mySQL on Amazon and use SQL Server there, which would make Baby Jesus cry.

Has anyone gotten mySQL to SQL Azure/SQL Server replication or syncing working? Or have any other approaches (a NoSQL solution that replicates and might meet our but-we-need-to-join-some-tables needs that can easily be run on Amazon and Azure)?

OK, so I took the stock Windows 2008 64-bit Amazon AMI and wanted to add a D: drive for page file space and crash dumps. I launched the instance with a second EBS volume attached as xvdf and went into Disk Management set it online, and added the page file and crash dump settings and all that works.

But when I reboot, the box comes back up with that second drive as "Offline." How do I get that disk to automatically come online on reboot (or most notably, when I turn this into an AMI and launch more instances off it - I've tried that too and same deal with the D:).

I have a Windows 2008 server with 7.5 GB of RAM and I want to get complete crash dumps off it.

I started by moving the page file mostly off the C: drive - I left a small 64-MB page file on the C: drive so there'd be one, but added a 8 GB initial and max page file to the D: drive using Control Panel/System/Advanced system settings/Performance/Settings.../Advanced/Virtual Memory/Change... I reboot, and I have an 8 GB pagefile.sys on my D: drive, all seems well.

Then I set up complete crash dumps, by setting HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\CrashDumpEnabled to 1 and change DumpFile to D:\Memory.dmp. I look under Control Panel/System/Advanced system settings/Startup and Recovery/Settings.../System Failure and I see the right settings. Then I provoked crashing using NotMyFault.

But I wasn't getting a crash dump. So I read this Microsoft doc and it says for 2008/Vista you have to add a DedicatedDumpFile entry in the registry. I added that, again at 8 GB, and now I get crash dumps, but I have both an 8 GB pagefile.sys AND an 8 GB dumpfile.sys on my D: drive (not to mention the 7.5 GB Memory.dmp dump file).

Am I missing something? Is there a way to use the same alternate page file for both normal paging and for crash dumps, so I'm not blowing 24 GB of space on all this?

I have a problem .NET 4 app I'm trying to load on my IIS7 server. It has a bug where if it gets hit multiple times while it's trying to come up - either initially or from a recycle - it gets deadlocked permanently.

I'm trying to figure out a way to either prevent traffic from coming to it while it's initializing, or in the worst case make the whole thing "single-threaded" so that it can never get two requests in at the same time. I'm already running just one worker process but that's not doing the trick. Ideas? Thanks...

I have a problem with a .NET 4 application leaking memory. It's not a "forgot to clean it up" leak in the normal sense, they are doing compiles that create a bunch of .NET assemblies in memory and they tell me there's no way to clear them out after. So it's now up to me to keep the service going.

So I'm faced with a couple options. There's the "set IIS worker processes to recycle when the memory gets large." That will help, but to me it seems like still an inherently metastable system, and there's user impact while the pool is recycling, isn't there?

They tried breaking out the compiles into separate temporary app domains, but then each operation is pretty slow, it adds ~15 seconds per hit to spawn the app domain. So they don't want to do that.

Is there anything other than "recycle a lot" I can do from the system side to mitigate this problem in a more permanent way? (Or app ideas, but that's more for SO.)

I'm having an odd problem where FreeSSHD doesn't start on boot about one in ten times.

I'm using it on Amazon EC2 instances actually, since there's a bug in Amazon's version of Xen which makes it impossible to use cygwin/openssh on their 64-bit instances.

So if I take one image of Windows Server 2008 Data Center with FreeSSHD installed and set to run automatically as a service, and spin up 40 instances, FreeSSHd doesn't start on around 4 of them. The service isn't started and there's nothing in the sshd log file.

In the Event Log I see:

A timeout was reached (30000 milliseconds) while waiting for the FreeSSHDService service to connect. The FreeSSHDService service failed to start due to the following error: The service did not respond to the start or control request in a timely fashion.

But there's nothing else wrong, it seems to just be intermittent. If I remote desktop in and start the service manually later, it runs fine.

Any tips here on how to mitigate this? I'm trying to automate provisioning of these systems, so I really need sshd to start reliably without manual intervention.

FreeSSHD isn't really maintained any more, so I suspect I need a more generic Windows answer to this. Increase the timeout? Is there a way to get it to try again?

We have a Java app I'm serving from Apache 2.2 + mod_proxy_ajp + Tomcat 6 that is a REST API and end users tend to like to totally slam it as fast as they can. I'd like to be able to throttle the hits per second, ideally by user/IP, to it. Like how Amazon, Twitter, etc. rate limits hits to their APIs.

1. Preferably in Apache or Tomcat itself, but in the OS (Fedora) is ok, although I don't want to affect the other Tomcat apps on the same box (so just doing iptables on port 80 isn't the best solution)
2. I don't want to limit bandwidth, this app doesn't produce more than 1k of output per hit, just the raw number of hits to the app to protect Tomcat from being overwhelmed
3. I'm running on Amazon AWS EC2 so no multiple IPs per box or other complicated networking solutions
4. I'm open to solutions I can tell the developers to put into their code, but would like it to be something I can control from an admin point of view (like I can change log levels using their log4j config files).

Seems like all the Apache modules to do stuff like that are all abandoned and not Apache 2.2 compatible. Ideas?

I'm configuring an Apache 2.2 front end to pass through specific apps to Tomcat via mod_proxy_ajp. Some of these apps, like the login/auth service, I want to enforce accepting HTTPS hits only.

My httpd-proxyajp.conf file has stanzas in it like this per app:

  ProxyPass /auth-1.0 ajp://localhost:8009/auth-1.0
  ProxyPassReverse /auth-1.0 ajp://localhost:8009/auth-1.0

  <Proxy /auth-1.0>
    Order Deny,Allow
    Allow from All
  </Proxy>

  <Proxy /auth-1.0/WEB-INF>
    Order Deny,Allow
    Deny from All
  </Proxy>

And I don't want to redirect http hits to them - it kinda defeats the purpose if someone writes a client that blindly passes their login credentials all the way to me in the clear and I just make them pass them encrypted a second time via a redirect. So I don't want to do the common solution,

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^/auth-1.0 https://%{HTTP_HOST}%{REQUEST_URI}

I really want to throw an error instead.

Complication: Doing this on Amazon EC2 so can't use IP-based virtual hosts and can't use name-based because I'm using SSL. I'd prefer to do this without vhosting anyway; I need the apps to be served off the same DNS name.

OK, I have a Windows 2008 Server scheduled task that runs on system startup. This works fine. But I want to move this task to another box (actually, I'm using Amazon and need this to work on other instances launched from this image but that's complexity not relevant to the problem).

So I export the task to XML, and try to import it using schtasks. But when I try to reimport it on the same box (just by changing the name), I get

C:\>schtasks /Create /XML "mytask.xml" /TN "My Task 2"

ERROR: No mapping between account names and security IDs was done.
(17,8):LogonType:

I look, and I notice that in the Principals section it says

 <UserId>IP-0AD68720\Administrator</UserId>

Which will make me sad when I try to move this task to another box. But if I edit that to just say "Administrator", when I import I get this error:

ERROR: Logon failure: unknown user name or bad password.

So I can't even reimport it on the same box, and can clearly see I'll have trouble importing it on another... Any hints? I want to alter the task so I can use it on other boxes to accomplish the same goal.

Thanks...

I'm working on getting an IPSec VPN working between Amazon EC2 and my on-premise. The goal is to be able to safely administer stuff, up/download data, etc. over that tunnel.

I have gotten the tunnel up in openswan between a Fedora 12 instance with an elastic IP and a Cisco router that's also NATted. I think the ipsec part is OK, but I'm having trouble figuring out how to route traffic that way; there's no "ipsec0" virutal interface because on Amazon you have to use netkey and not KLIPS for the vpn. I hear iptables may be required and I'm an iptables noob.

On the left (Amazon), I have a 10. network. Box 1 is privately 10.254.110.A, publically IP 184.73.168.B. Netkey tunnel is up. Box 2 is publically 130.164.26.C, privately 130.164.0.D

And my .conf is:

conn ni
        type=           tunnel
        authby=         secret
        left=           10.254.110.A
        leftid=         184.73.168.B
        leftnexthop=    %defaultroute
        leftsubnet=     10.254.0.0/32
        right=          130.164.26.C
        rightid=        130.164.0.D
        rightnexthop=   %defaultroute
        rightsubnet=    130.164.0.0/18
        keyexchange=    ike
        pfs=            no
        auto=           start
        keyingtries=    3
        disablearrivalcheck=no
        ikelifetime=    240m
        auth=           esp
        compress=       no
        keylife=        60m
        forceencaps=    yes
        esp=            3des-md5

I added a route to box 1 (130.164.0.0/18 via 10.254.110.A dev eth0) but that doesn't do it for predictable reasons, when I traceroute the traffic's still going "around" and not through the vpn.

Routing table:

10.254.110.0/23 dev eth0  proto kernel  scope link  src 10.254.110.A
130.164.0.0/18 via 10.254.110.178 dev eth0  src 10.254.110.A
169.254.0.0/16 dev eth0  scope link  metric 1002

Anyone know how to do the routing with a netkey ipsec tunnel where both sides are NATted?

Thanks...