M. Glatki's questions

for a Domain mydomain.com, I have the following entries:

; Canonical Name definition
*            600    IN CNAME    www
*.staging    600    IN CNAME    newdomain.com.
www                 IN CNAME    olddomain.com.

My expectation would be, that a.mydomain.com would always resolve to whatever is behind olddomain.com, and that a.staging.mydomain.com would always resolve to whatever is behind newdomain.com.

Sadly, my results are not consistent, although all TTLs have been expired a week ago. Sometimes i get the expected results:

$ nslookup abc.staging.mydomain.com

Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
abc.staging.mydomain.com canonical name = newdomain.com.
Name:   newdomain.com
Address: 2.2.2.2

Sometimes I don't:

$ nslookup abc.staging.mydomain.com

Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
abc.staging.mydomain.com canonical name = www.mydomain.com.
www.mydomain.com.   canonical name = olddomain.com
Name:   olddomain.com
Address: 1.1.1.1

How can I configure my DNS zone to achieve what I want?

I added a VPC endpoint to my VPC using CloudFormation, and allowed s3 usage. The routes are visible in the AWS console, but not in the local routing tables of the EC2 instances:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.29.4.129    0.0.0.0         UG    0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
172.29.4.128    0.0.0.0         255.255.255.128 U     0      0        0 eth0

How do I verify that the EC2 instances in the VPC actually uses the VPC endpoint for S3, and not the available internet connection?

I am currently trying to figure out a good configuration to make a Bastion host highly available. I want to meet the following targets:

1. The bastion host(s) need to able to withstand a Availability Zone failure and ec2 instance failure. A small downtime (a few minutes) may be acceptable.
2. The bastion host(s) needs to be reachable via a permanent DNS entry.
3. No manual intervention needed

My current setup is as follows: Bastion host in Auto Scaling Group in two availability zones, ELB in front of the Auto Scaling Group.

This setup has a few advantages:

• Easy to setup using CloudFormation
• Auto Scaling Groups over two AZs can be used to guarantee availability
• The does not count towards the accounts EIP limit

It also has some disadvantages:

• With two or more bastion hosts behind the ELB, SSH host key warnings are common, and I do not want our users to get accustomed to ignore SSH warnings.
• The ELB costs money, as opposed to EIP. About as much as the bastion host, actually. This is not really much of a concern, I added this point only for sake of completeness.

The obvious other solution is to use an ElasticIP, which has - as I see it - a few drawbacks:

• I can (obvously) not attach an EIP to an Auto Scaling Group directly
• When not using Auto Scaling Groups, I have to put something in place that starts new EC2 bastion hosts if the old ones fail, e.g. using AWS Lambda. This adds additional complexity.
• When the EIP is attached to an Auto Scaling Group manually, on Availability Zone failure, the EIP will get unattached and not be reattached to a new instance. Again, this can be resolved by running a program (on the instance or AWS Lambda) that reattaches the EIP to an instance. Again this adds additional complexity.

What are best practices for High availability SSH instances, i.e. bastion hosts?

I am running Ansible on an EC2 instance with an assigned Iam role. I am running this playbook:

$ cat s3.yaml
---
- hosts: localhost
  remote_user: ec2-user
  tasks:
    - name: download ec2.py from s3
      s3:
        bucket: mybucket
        object: /ec2.py
        dest: /tmp/ec2.py
        mode: get

Running it with -vvv provides this error message:

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "aws_access_key": null, 
            "aws_secret_key": null, 
            "bucket": "mybucket", 
            "dest": "/tmp/ec2.py", 
            "ec2_url": null, 
            "encrypt": true, 
            "expiry": "600", 
            "headers": null, 
            "marker": null, 
            "max_keys": "1000", 
            "metadata": null, 
            "mode": "get", 
            "object": "/ec2.py", 
            "overwrite": "always", 
            "permission": [
                "private"
            ], 
            "prefix": null, 
            "profile": null, 
            "region": null, 
            "retries": 0, 
            "rgw": false, 
            "s3_url": null, 
            "security_token": null, 
            "src": null, 
            "validate_certs": true, 
            "version": null
        }, 
        "module_name": "s3"
    }, 
    "msg": "Source bucket cannot be found"
}

According to documentation, Ansible with boto should be able to pick up the EC2 instance role.

So far I have:

• Tried the documentation, which implies that it should just work.
• Verified my boto version is new enough (boto 2.42)
• Verified the EC2 instance role has the correct permissions (aws s3 cp s3://mybucket/ec2.py /tmp/ec2.py works just fine)
• Verified the EC2 instance credentials are available through curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

This answered question and the documentation indicates that it is possible.

Can I accomplish this without making the instance credentials available to boto/ansible directly? If yes, how. The documentation seems a bit lacking.

I am currently running Puppet with in masterless mode. I am using r10k for module and environment deployment.

Simplified version: The r10k control repository has two branches: testing and production. Changes in production will be automatically distributed to production servers. Changes in testing to some staging servers.

Now, if I am changing things in testing, I sometimes have to change the r10k control repository, too. A common example would be the Puppetfile, which currently looks like this in production:

forge 'forge.puppetlabs.com'

# Forge modules
mod 'puppetlabs/stdlib'
mod 'puppetlabs/concat'
mod 'saz/ssh'

# Custom modules
mod 'ownmodule1',
        :git => 'https://git.example.org/configuration/ownmodule1.git',
        :ref => 'production'
mod 'ownmodule2',
        :git => 'https://git.example.org/configuration/ownmodule2.git',
        :ref => 'production'

The configuration for the Custom modules might look like this on the testing branch:

mod 'ownmodule1',
        :git => 'https://git.example.org/configuration/ownmodule1.git',
        :ref => 'testing'
mod 'ownmodule2',
        :git => 'https://git.example.org/configuration/ownmodule2.git',
        :ref => 'testing'

Now, a commit in testing might look like this:

+mod 'ownmodule3,
+        :git => 'https://git.example.org/configuration/ownmodule3.git',
+        :ref => 'testing'

If I merge this to production, and are not careful, ownmodule3 will be added to production with the testing branch, which could be fatal. This also prevents automated merging when all tests are successful.

How can I modify my repositories or workflow to prevent the accidental merging of branch specific changes?

I have several crontabs defined in a pp file, e.g:

cron { 'puppet-cron1':
    ensure  => present,
    command => "cat /etc/issue",
    user    => root,
    minute  => '*/30',
}
cron { 'puppet-cron2':
    ensure  => present,
    command => "cat /etc/issue",
    user    => www-data,
    minute  => '*/30',
}

How do I list all cronjobs created by puppet on a system?

I tried puppet ressource cron, which lists only the root cronjob:

cron { 'puppet-cron1':
  ensure  => 'present',
  command => 'cat /etc/issue',
  minute  => ['*/30'],
  target  => 'root',
  user    => 'root',
}