Garreth McDaid's questions

I currently use 2 DNS providers to publish my zone, for the purposes of redundancy. To cut costs/complexity/overhead, I'd like to use a single provider, but I'd still like some redundancy in the event of protracted outages. I also have my own local Bind server on which I publish my zone, but this is not connected to the public internet.

I was thinking I could do the following:

Add an NS records for my zone that points to an ip address that maps to my local Bind server. Under normal circumstances, this IP address would not respond to DNS queries, but in the event there was an outage on my main provider, I could open up this connection and my local Bind server would start responding. When the outage had cleared, I would then roll back the connection.

My understanding of recursive DNS is that local DNS systems will monitor the RTT of authoritative DNS servers and establish affinity with the ones that respond the fastest. If that were true of most local DNS systems, my "phantom" NS server should not receive very many queries while it was offline, which would mitigate any impact on the end user of having such a system.

Is this a reasonable/sensible approach to DNS redundancy?

As per documentation:

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html

If you want to use ebextensions with a Single Container Docker Elasticbeanstalk environment, you have to:

1. Use a zip archive as your application version
2. Include a Dockerfile.aws.json file in the archive
3. Include a Dockerfile file in the archive
4. Include a .ebextensions folder in your archive

The Docker image used will be the image created by the Dockerfile, and not the image specified in Dockerfile.aws.json

In my case, I don't want Elasticbeanstalk to build my Docker image on the fly. Instead, I want to use an image from my AWS ECR.

In that case, can I just specify a Dockerfile that only includes:

FROM F11111111111.dkr.ecr.us-west-2.amazonaws.com/<my-image>:latest
EXPOSE 80

Will this just build a local non-modified image from the base and use that?

My company has reached that tipping point where what we are spending on EBS storage warrants a serious look at moving our storage to a SAN/NAS in a Co-Lo location connected with AWS Direct Connect.

Pricing Co-Lo and Connectivity in this regard is straightforward, but pricing hardware is very complex at this early evaluation stage.

1TB of storage in AWS EBS costs $1,200 per annum ($0.10 per GB per month).

In the order of 300TB, and assuming 3 year depreciation, is it reasonable to think we can get Co-Lo SAN/NAS for < $3,600 per TB?

We don't need All-Flash. IOPS requirement would be about 2k IOPS IN+OUT.

I'm trying to centralise logging in an environment that using multiple application technologies (Java, Rails and various DBs).

We want to developers to bring up stacks with Docker Compose, but we want to them to refer to a central log source (ELK) to debug issues, rather than trying to open shells into running Docker containers.

The applications all write to the file system rather than to STDOUT/STDERR, which removes all of the options associated with the Docker logging driver, and logspout too.

What we have done is configure the containers to have rsyslog include the application log files and forward those to logstash which has a syslog input. This works in terms of moving the logs from A to B, but managing multi-technology logs in ELK based on the syslog input is horrible (eg trying to capture multine Java stacktraces, or MySQL slow queries).

Is there a better way to do this? Should I be running logstash in each container, so that I can apply filters and codecs directly to the log files, so that I don't have to rely on the syslog input?

Of is there some way to use the Docker logging driver with application log files that are written to the file system?

The logstash documentation indicates that you can collapse the multiple indented lines in a Java stacktrace log entry into a single event using the multiline codec:

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

input {
   syslog {
       type => syslog
       port => 8514
       codec => multiline {
            pattern => "^\s"
            what => "previous"
       }
  }
}

This is based on logstash finding an indent at the start of the line and combining that with the previous line.

However, the logstash documentation is the only place where I can find a reference to this. The general user community seems to be using elaborate grok filters to achieve the same effect.

I've tried the basic indentation pattern provided by logstash, but it doesn't work. Has anyone else managed to get this working by matching the indentation pattern?

There are various ways to trigger a Jenkins job from an SCM like Bitbucket, but what I want to do specifically is trigger a build using the branch that is the source of the Pull Request.

Up to now, we have used the Bitbucket Pull Request Builder, but it is very flaky and unreliable, and not supported well.

https://wiki.jenkins-ci.org/display/JENKINS/Bitbucket+pullrequest+builder+plugin

Bitbucket do supply quite good features in terms of Webhooks, which when used with the Jenkins Git Plugin, do allow for triggering of builds based on various Bitbucket events (eg a Pull Request update).

There is also the Bitbucket Webhook plugin, but again that doesn't offer much in terms of dynamically choosing the branch you want to build.

https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin

However, what this seems to do is trigger a poll of the repo, where is then tries to build any branch that is different from the main branch.

Our use case is that we allow developers create their own branches, for which they then create Pull Requests to the development branch.

There doesn't seem to be any way to trigger a build that uses the developer created branch as the build branch (other than the aforementioned Bitbucket Pull Request Builder).

Am I right or wrong in this?

We are in the process of moving our MySQL DB from AWS RDS to EC2.

RDS provides us with with a solid HA solution, which we want to replicate in EC2. We've looked at MySQL Utilities and MHA, both of which present problems for us. As a result, we're thing about a DIY solution, which will work as follows:

We will have a Master and a Slave replica (Slave 1). We will have a Slave replica of the Slave (Slave 2).

Master and Slave 1 will be behind a HA Proxy cluster.

We will monitor the Master very closely.

If the Master monitor detects a failure, we will run a script that shuts down the Master EC2 instance and stops the Slave process on the replica.

Slave 1 will now become the Master, and Slave 2 will become Slave 1. The new Slave 1 will be added to the HA Proxy cluster.

We will then rebuild Slave 2 from a backup of Master (backup every 3 hours).

We obviously need to be careful to ensure that all steps in the process exit cleanly, and that we have rock-solid error handling.

Can anyone offer an opinion as to the viability of our proposed DIY solution?

I am using the Logstash S3 Input plugin to process S3 access logs.

The access logs are all stored in a single bucket, and there are thousands of them. I have set up the plugin to only include S3 objects with a certain prefix (based on date eg 2016-06).

However, I can see that Logstash is re-polling every object in the Bucket, and not taking account of objects it has previously analysed.

{:timestamp=>"2016-06-21T08:50:51.311000+0000", :message=>"S3 input: Found key", :key=>"2016-06-01-15-21-10-178896183CF6CEBB", :level=>:debug, :file=>"logstash/inputs/s3.rb", :line=>"111", :method=>"list_new_files"}

ie

Every minute (or whatever interval you have set) Logstash starts at the beginning of the bucket and makes an AWS API call for every object it finds. It seems to do this to find out what the last modified time of the object is, so that it can include relevant files for analysis. This obviously slows everything down, and doesn't give me real time analysis of the access logs.

Other than constantly updating the prefix to match only recent files, is there some way to make Logstash skip reading older S3 Objects?

There is a sincedb_path parameter for the plugin, but that only seems to relate to where the data about what file has last been analysed is written.

I have 3 OpenSSH servers and I want to have only one.

I have created a new server, and migrated all the accounts/data from the other 3 onto the new server.

I now intend to update the DNS records for the 3 existing servers so that they point to the ip address of the new server.

However, when clients who connected to any of the older servers now connect to the new server, they are going to get an error, as the server key in their known hosts file is going to differ from the server key issued by the new server.

I can move the server key from one server to the new server, so that there are no issues with one of the three, but there are still going to be problems with the other two.

I had thought about attached 2 extra ip addresses to the new server, and running 3 separate SSH daemons on it, with different server keys, but I'd really like to avoid this.

Has any one got any other suggestions.

I have a farm of production mysql servers that are currently protected by restricted passwords.

I would prefer to restrict access to these systems at network level, but have to account for the fact that some members of the dev community require read only access to debug applications issues.

I know that I can do this by creating mysql users that can authenticate at different levels from different ip subnets (eg 'user'@'10.0.0.8') but I'd like to avoid having to refactor code to introduce new users into the application (which would require a considerable QA effort).

Ideally, I just want to break the network link between where the mysql servers are (in a dedicated subnet) and the developers are (in a dedicated subnet) and allow the application servers continue to access the mysql servers as before.

It would seem like some sort of TCP based proxy with authentication would suit this purpose. In that way, I could create user specific accounts on the proxy that would create an audit trail and provide short-term on demand access to developers without having to tinker with the user database on the MySQL servers.

However, authentication options in software like HAProxy and MySQL proxy appear quite limited.

Has anyone implemented something similar?

I am implementing a reverse http proxy:

proxy.example.com

This will forward requests to servers based on URI:

proxy.example.com/server1 -> server1.example.com

When a user requests proxy.example.com/server1, server1 sends a programatically generated (Ruby Devise Gem) 302 response with the following "Location" value:

proxy.example.com/users/sign_in

I need this to be:

proxy.example.com/server1/users/sign_in

I have implemented the following config in Apache:

ProxyPass "/server1/" "http://server1.example.com/"
ProxyPassReverse "/server1/" "http://server1.example.com/"

as per:

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypassreverse

"This directive lets Apache adjust the URL in the Location, Content-Location and URI headers on HTTP redirect responses. This is essential when Apache is used as a reverse proxy (or gateway) to avoid bypassing the reverse proxy because of HTTP redirects on the backend servers which stay behind the reverse proxy."

But the Location header that is being returned by server1 is still:

proxy.example.com/users/sign_in

Is there something wrong with my config?

thx

My colleague and I use Ansible to manage a group of servers.

We use the .ansible.cfg file in our local home directories to setup our local environments and keep our playbooks in a git repo.

When authenticating to servers, I use user1, and he uses user2. 95% of of our servers have these accounts, but historically reasons, a few servers only have a "user" account.

We're using host_vars to set the remote_user variable for the minority of servers in question.

However, in our playbooks, we generally user "all" to stipulate what servers we want to hit, and use the --limit parameter on the command line to specify exactly which servers should get the update. Our server farm is a legacy of mis-mash poorly engineered servers that have to be kept online until they are retired in a few years, and we've found that this approach best suits our needs.

Our issue is that our remote_user parameter is set in our .ansible.cfg file, where it is exposed as environment variable rather than a script variable.

That means if our task contains:

remote_user: "{{ remote_user }}"

It will only work for hosts for which that variable is defined

For the 95% of hosts for which we don't define this variable, the task fails.

Is there a way to only use the variable if it is defined?

eg

If remote_user is defined, use it, if not, use the environment variable set in .ansible.cfg

Note: I know I can use:

- name: Do something 
  remote_user: "{{ remote_user }}"
  when: remote_user is defined

In a task definition, but that will only apply to that task, and I don't want to have to update all task

When I really need is for that condition to be available at the hosts definition, ie:

---
- hosts: all
    remote_user: "{{ remote_user }}"
    when: remote_user is defined

But that is illegal in Ansible

I am currently migrating a DNS zone from one DNS server provider to another. I am trying to estimate how long it will take for the change to propagate, and to understand what the delay might be if I chose to rollback mid-stream.

Previously, I thought I could do:

dig example.com ns

To see what the remaining TTL on the NS record was, but now I understand that this NS record is the NS record for subdomains in the zone, and not the NS record that emanates from the root servers, which is the one that ultimately determines to which name server the query will be sent.

I tested this by setting up a test record in the zone in each of the providers:

Provider1 test.example.com 10.0.0.1
Provider2 test.example.com 192.168.0.1

For both providers, the TTL on the NS records in 0, while the NS records at the TLD Registrar level point to the name servers of Provider1.

When I change the NS records in the zone at Provider1, I can see this reflected in NS queries almost immediately (using 'dig example.com ns').

However, when I send a query for an A record, ie

test.example.com

it always returns

10.0.0.1

regardless of what the NS records in the zone at Provider 1 are set to.

On this basis, I've concluded that the NS records within the zone file are irrelevant to the migration, and that only the name servers records at the TLD level are important.

However, I can't get a read on how long it is likely for a change there to propagate, either forward or back.

Is it possible to query what TTL are am working with for records emanating from the TLD root servers?

I have a server that relies on puppet vcsrepo to update code in a local mercurial repository based on a tag.

When I change the required tag, using the vcsrepo "revision" parameter, vcsrepo should do a hg pull and hg update on the repo.

This is all working fine.

However, I have created a clone of this server to test something else, and now when I run the puppet update I get an error:

Not trusting file /var/hg/repo/.hg/hgrc from untrusted user *user*, group *group*

This happens because puppet is running as root, while the hgrc file is owned by user

The user parameter in vcsrepo is supposed to deal with this:

vcsrepo { '/var/hg/repo':
    ensure   => present,
    provider => hg,
    source   =>       'ssh://****',
    user => 'user',
    owner => 'user',
    group => 'group',
    revision => '1.12'
  }

ie

the hg commands are supposed to run as user so that the Trust requirement in mercurial is satisfied.

But its not working. The clone server is a bit for bit copy of the original.

I have inherited an Amazon AWS environment in which the Root account key has been widely distributed for the purposes of making backups to S3 buckets.

I need to track down where the key is being used, so that I can replace it with a key with limited permissions.

I've setup event notifications on the buckets to send messages to an SQS Q to tell me when objects are created. These messages contain the ip address of the server from where the object request originated, but not the IAM key in use (only the ID of the Amazon Account in use).

Cloudtrail isn't of use here either, as S3 requests are not written to Cloudtrail.

Is there any other way I can tell which key is being used when making S3 requests?

PLEASE NOTE THAT THIS IS NOT A SUGGESTED SOLUTION, MERELY AN UPDATE ON WHAT I DID. USE THE METHOD SUGGESTED ABOVE. I SHOULD HAVE UPDATED THE QUESTION.

Thanks EEAA

I considered that, but there are hundreds of buckets in play, so it wasn't really practical. I was hoping there was somewhere in IAM that I could track down key usage.

Given that I had Events and SQS already set up, what I ended up doing eventually was to set up events on likely buckets and match the S3 event timestamp to the timestamp provided by the key-last-used timestamp from IAM.

This gave me the ip addresses of the servers that were sending requests to S3 at the same time as the Root key was being used, out of which I was able to find the root key on a few servers. Hopefully, when I check the root key over the next few days, I will no longer see it in use.

If not, I will probably have to set up logging on individual buckets as you've suggested.

The RabbitMQ clustering document

https://www.rabbitmq.com/clustering.html

describes a process whereby you can deploy a configuration file for clustering so that a cluster is automatically created when your rabbitmq nodes boot.eg

[{rabbit,
  [{cluster_nodes, {['rabbit@rabbit1', 'rabbit@rabbit2', 'rabbit@rabbit3'], ram}}]}].

You apply this file on each of the 3 nodes at /etc/rabbitmq/rabbitmq.config, and then start rabbitmq on each and a cluster will apparently form.

This doesn't appear to work too well eg

If you start rabbit2, and rabbit3 hasn't already come up, the service will not start on rabbit2, as it is trying to create a cluster with rabbit3.

Equally, if you only apply the config on rabbit1, and have pre-started rabbit2 and rabbit3, rabbit1 will form a cluster with rabbit2 only, as, according to the documentation (I don't understand why):

RabbitMQ will try to cluster to each node provided, and stop after it can cluster with one of them.

The only way this seems to work is if you apply the config file on all 3 nodes, and have them start at exactly the same time. I am trying to deploy this with Ansible, which creates the nodes sequentially, so that doesn't work.

What am I missing here?

I want to create an IAM policy that allows a user deploy instances as follows:

1. They can only use 1 AMI
2. They can only deploy to 1 specific VPC subnet
3. They can only use 1 specific VPC security group

This scenario is addressed in the VPC documentation here (Example 4):

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_IAM.html#subnet-sg-example-iam

I have tried my own version of the policy as such:

{
"Version": "2012-10-17",
"Statement":[{
    "Effect":"Allow",
    "Action": "ec2:RunInstances",
    "Resource": [
        "arn:aws:ec2:eu-west-1:937821706121:image/ami-141ac363",
        "arn:aws:ec2:eu-west-1:937821706121:subnet/subnet-733de516",
        "arn:aws:ec2:eu-west-1:937821706121:network-interface/*",
        "arn:aws:ec2:eu-west-1:937821706121:volume/*",
        "arn:aws:ec2:eu-west-1:937821706121:key-pair/*",
        "arn:aws:ec2:eu-west-1:937821706121:security-group/sg-4aa80f2f"
    ]
}]
}

It doesn't work. I get permission denied when I attempt to deploy instances as a user who is a member of a group where this policy applies. Is there some other policy I need to include with this to allow instance deployment in this way?

I have an Amazon VPC set up between AWS and my corporate LAN.

Originally, when I set it up, I specified static routing for the following CIDRs:

10.55.55.0/24 10.77.77.0/24 10.99.99.0/24

I set up the following CIDR in the VPC:

10.44.0.0/16

This all worked fine.

I now have a new network on my Corporate LAN: 10.55.54.0/24

I added a new route in the main routing table for this, pointing to the same vgw as the others.

When I know look at the routes, I see that the original 3 routes are propagated, but the 4th new one is not.

On a server instance in 10.44.0.0/16, I can still connect to hosts in the existing 3 CIDRs, but I am not seeing any routing for the new CIDR:

[server]$ traceroute 10.55.55.88
traceroute to 10.55.55.88 (10.55.55.88), 30 hops max, 60 byte packets
1  169.254.254.37 (169.254.254.37)  0.543 ms  0.524 ms  0.504 ms
2  169.254.254.5 (169.254.254.5)  1.278 ms  1.007 ms  0.985 ms
3  169.254.254.2 (169.254.254.2)  32.596 ms 169.254.254.6 (169.254.254.6)  35.244 ms     169.254.254.2 (169.254.254.2)  32.830 ms
4  ip-10-55-55-88.eu-west-1.compute.internal (10.55.55.88)  36.822 ms  37.338 ms  36.842 ms


[server]$ traceroute 10.55.54.88
traceroute to 10.55.54.88 (10.55.54.88), 30 hops max, 60 byte packets
<timeout>

What step am I missing here? Do I have to recreate the entire VPC/VPN to include the new CIDR?

I use elasticsearch as part of a Logstash stack, in which all of the components of the stack are installed on the same server.

The purpose of this is to expose application logs to developers for debug purposes. I don't need to keep the indices created. I have a cron job that removes indices that are older than 7 days.

The raw logs are preserved elsewhere in case we need historical analysis.

The problem I have is that elasticsearch keeps entering a Red health state due to unassigned shards. I've researched various ways to recover this, but inevitably, I end up deleting the raw index files and restarting the service.

This is a real pain, as its always the time when the developers need access that elasticsearch is borked.

It seems add to me that there isn't an easier way to recover elasticsearch other than to delete the offending indices. I've configured elasticsearch to use a single node, no replicas and not to do any network discovery, but every couple of days, it keeps falling over.

Am I wasting my time trying to run elasticsearch on single server? Is it always going to keep falling over due to unassigned shards? Given what I use it for, it would seem like overkill to actually have to deploy a cluster.

Note: I am running this stack in Amazon EC2

I have been managing the IT function in a small software development company that has grown from 10 users to 70 users in 5 years.

Most users in the company are office based, but some key users work remotely.

At the beginning, Skype become the de facto tool for communicating between office bound and remote workers. As the company has grown, and more remote workers have been added Skype has become part of the ecosystem of the company.

In the last 18 months Skype reliability has deteriorated. I had initially attributed this to issues with a certain ISP, but in the interim we have moved offices contracted with a completely different ISP, increased our synchronous fibre broadband (20mbps), and installed new network hardware.The Skype issues however remain.

The standard complaint form remote users is that when they talk to their friends, mothers, wives, etc Skype is fine, but when they talk to someone in the office, they frequently get drops, pixelation, and muffling.

After having tried to resolve this over 2 years I'm beginning to wonder if Skype is an appropriate tool for this environment. All of the company's infrastructure is cloud based, so there is a constant data flow in and out of the office which will obviously peak from time to time. We're also Apple based, so there is constantly stuff downloading from iTunes often on multiple systems at the same time.

Also, there could be multiple Skype VC calls going on at the same time. We generally see issues in the morning between 09:00 and 11:00 when we have stand up meetings all of which involve remote developers.

Whenever we get these problems we increase the broadband by 5mbps which makes things better for a few months then we get some more staff, and then it starts again.

My theory is that these inevitable peaks which you generally won't get on a residential connection, are impacting on Skype audio and video. With the company IT profile as is making something like Skype work consistently is going to be virtually impossible. Maybe a dedicated internet connection for Skype would be a solution?