Philip Couling's questions

I'm trying to allow VPN users to use Private Route53 DNS entries. Eg: a private hosted zone example.corp issuing DNS records on the local network.

At the moment I can't even find the right set of configurations to allow connected clients to use amazon's DNS at all.

Current (simplified) setup

The client VPN endpoint is allocated CIDR range 172.20.0.0/22. It has one configured target network association which has CIDR 172.20.254.0/24.

The target network is empty; meaning no EC2 instances live there etc.. However it does itself have onward routing rules to allow traffic to other networks. Access to those onward routes is controlled through "Authorization Rules". The route table has just one entry stating 172.20.0.0/16 (the whole VPC) is forwarded to the target network association.

The security group associated with the VPN is very permissive allowing anything on 172.20.0.0/16.

This configuration has worked without DNS for over a year. The only changes I'm now making are to enable DNS.

Changes - Attempting to enable DNS

This page tells me that the DNS should be available on the network range +2. I've tried setting the VPN DNS server to 172.20.0.2 and 172.20.254.0 (not at the same time). But neither respond to DNS request or ICMP ping.

In an attempt to get this working I have tried:

• Ensured that "DNS resolution" and "DNS hostnames" were both set on the VPC
• Adding Authorization Rules for both 172.20.254.0/24 and 172.20.0.2/22.
• Adding route table entry for 172.20.0.0/22 to local

How am I supposed to setup Client VPN Endpoint to use AWS DNS?

We have a situation where we have multiple EC2 instances each running a VPN. Both the remote VPN server and remote subnet are run by a third-party and we have no say in the way they are setup.

We don't believe these are transferable to AWS client Lan-to-Lan VPN

The VPNs all route to the same physical subnet with the same CIDR block. There is some rate limiting for these VPNs (on the remote side) and we don't want to push all of our traffic for that CIDR block through the same subnet. Besides this we would like to have some form of health-check and fail-over so that if one VPN connection goes bad, we can re-route through another.

Does AWS have any form of transparent load-balanced routing? ...as opposed to an application load balancer. Likewise I believe AWS's Network load balancer acts as an endpoint routing specific ports to multiple providers.

Just to make this more complex, the VPN clients include a NAT meaning that the routing would need to be stateful.

This is something I'm aware is available on enterprise level hardware (cisco routers etc.) but I'm not sure if Amazon exposes any such feature.

I'm searching for a way to add a domain name to an ECS service (containers) without a load balancer.

The particular type of service I want to do this for is itself a load balancer with capabilities outside of AWS's offering. We need this to be redundant across availability zones with similar setups in multiple isolated VPCs. I'm really loathed to spend $2000 per year to put in enough loadbalancers to fulfil this simple requirement.

To be clear I'm looking for a solution which resolves an FQDN to an external IP address attached to the container so A or AAAA name (possibly CNAME if its weighted and points to an A or AAAA). It doesn't matter to me which network mode this works for. The container can be placed in any of host, bridge, awsvpc if the solution works!

So far I've found this frustratingly limited:

• Service Discovery will only add SRV records for host and bridge type network containers
• Service Discovery will create A and AAAA records for awsvpc network type containers, but awsvpc tasks can't be placed on ECS EC2 instances and still have a public IP address.
• Running the same tasks in fargate would cost as much or more than the load balancers.
• Route53 has no good mechanism for adding A and AAAA name records to EC2 instances in an autoscaling group so even if I put an instance of this task on every node I still won't be able to reference it by domain name.

How to add a domain name to an ECS container without a load balancer?

From what I read here ECS capacity providers should (generally) prevent tasks immediately failing on resource limits by putting them in a "Provisioning" state and spinning up a new EC2 instance

This means, for example, if you call the RunTask API and the tasks don’t get placed on an instance because of insufficient resources (meaning no active instances had sufficient memory, vCPUs, ports, ENIs, and/or GPUs to run the tasks), instead of failing immediately, the task will go into the provisioning state (note, however, that the transition to provisioning only happens if you have enabled managed scaling for the capacity provider; otherwise, tasks that can’t find capacity will fail immediately, as they did previously).

I've setup an ECS cluster, with autoscaling group and ECS capacity provider in terraform. The autoscaling group is set with min_size = 1 and it immediately spins up a single instance... so I'm confident my launch configuration is fine.

However when I repeatedly call "RunTask" through the API (tasks with memory=128), I get tasks failing to start immediately with reason RESOURCE:MEMORY. Also no new instances start up.

I can't figure out what I've misconfigured.

This was all setup in terraform:

resource "aws_ecs_cluster" "ecs_cluster" {
  name = local.cluster_name


  setting {
    name  = "containerInsights"
    value = "enabled"
  }
  tags = var.tags
  capacity_providers = [aws_ecs_capacity_provider.capacity_provider.name]

  # I added this in an attempt to make it spin up new instance 
  default_capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.capacity_provider.name
  }

}

resource "aws_ecs_capacity_provider" "capacity_provider" {
  name = "${var.tags.PlatformName}-stack-${var.tags.Environment}"

  auto_scaling_group_provider {
    auto_scaling_group_arn         = aws_autoscaling_group.autoscaling_group.arn
    managed_termination_protection = "DISABLED"

    managed_scaling {
      maximum_scaling_step_size = 4
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 100
    }
  }

  tags = var.tags
}

#Compute
resource "aws_autoscaling_group" "autoscaling_group" {
  name                      = "${var.tags.PlatformName}-${var.tags.Environment}"
  # If we're not using it, lets not pay for it
  min_size                  = "1"
  max_size                  = var.ecs_max_size
  launch_configuration      = aws_launch_configuration.launch_config.name
  health_check_grace_period = 60
  default_cooldown          = 30
  termination_policies      = ["OldestInstance"]
  vpc_zone_identifier       = local.subnets
  protect_from_scale_in     = false

  tag {
    key                 = "Name"
    value               = "${var.tags.PlatformName}-${var.tags.Environment}"
    propagate_at_launch = true
  }

  tag {
    key                 = "AmazonECSManaged"
    value               = ""
    propagate_at_launch = true
  }

  dynamic "tag" {
    for_each = var.tags
    content {
      key = tag.key
      propagate_at_launch = true
      value = tag.value
    }
  }

  enabled_metrics = [
    "GroupDesiredCapacity",
    "GroupInServiceInstances",
    "GroupMaxSize",
    "GroupMinSize",
    "GroupPendingInstances",
    "GroupStandbyInstances",
    "GroupTerminatingInstances",
    "GroupTotalInstances",
  ]
}

I'd like to setup a docker service (docker-compose) on a Linux host with a container mounting an entire [removable] physical hard drive as a docker volume.

I know that it's trivial to setup bind mounts in docker-compose and I can manually mount the drive on the host. That approach does work, but it involves manual steps with an opportunity for human error. Bad things happen when this service starts with a blank directory on the host's drive root partition. Worse things happen if the drive is unplugged while still mounted.

What I'd like to do is have docker mount the device directly as a volume. This would have the advantage of fewer / simpler manual steps and a failsafe that if the drive was missing the service would fail to startup. This also would ensure the drive us unmounted when the service is stopped.

Given that volumes are basically just OS mounts, it feels like this should be simple, but numerous searches through the documentation and I'm still no further forward.

I'm struggling to get my head round port ranges described by AWS Security Groups and how they behave. I'm a software developer with many years of experience writing networking software so it's possible I'm just overcomplicating things.

What I'm really looking for is a way to tie up my networking knowledge with the naming and behaviour of AWS.

What confuses me the most is that on TCP or UDP packet there are two port numbers (the sender's and the recipient's). And this applies wether the packet is being send from server to client or client to server. So firewall rules can theoretically refer to 4 different ports:

1. Inbound packet recipient port
2. Inbound packet sender port
3. Outbound packet recipient port
4. Outbound packet sender port

I realise that any (TCP) connection will in practice have only two ports because inbound and outbound packets are mirrors of each other.

Now with all that in mind, the AWS console has just one port for inbound rules and one port for outbound rules. And when I look for examples they often include reference to allowing every port outbound on the same group as a single port inbound (see terraform example).

What precisely does this do?

I'm worried because... I seem to need this egress rule. This seems to allow any client to connect to port 443, from any port. (That's fine). But can the egress rule from one security group be combined with the ingress rule from another?

Eg: if I add a network group to let my machine act as a HTTP(S) server, like the example, and then I add another rule to let it act as a HTTP(S) client, then I will have one rule allowing any port from anyware and one rule allowing any port to anywhere. Does that completely open up the firewall or must each packet match a security group completely?

I'm trying to run RabbitMQ in ECS on an AWSVPC network with EFS persistence.

The problem is that ECS on AWSVPC resets the hostname every time the container is restarted. This results in rabbitmq starting an entirely new directory in /var/lib/rabbitmq so it loses all of it's persistence.

Eg: starting it once, /var/lib/rabbitmq will contain:

drwxr-xr-x  4 100 root 6.0K Dec 23 13:00 rabbit@ip-10-0-12-171
-rw-r--r--  1 100 root   90 Dec 23 12:57 rabbit@ip-10-0-12-171-feature_flags
drwxr-xr-x 10 100 root 6.0K Dec 23 12:57 rabbit@ip-10-0-12-171-plugins-expand

The next time it starts the IP will change and everything else will just be dropped.

I've tried setting the HOSTNAME environment variable but that has no effect whatsoever.

I've tried setting RABBITMQ_MNESIA_DIR but after the second start the container just cycles and shows no obvious error message for me to post here.

I've tried setting [email protected] (a route53 managed domain name) on the container. But that results in a puzzling error:

   [error] Supervisor net_sup had child net_kernel started with net_kernel:start_link(['[email protected]',shortnames], false, net_sup_dynamic) at undefined exit with reason {'EXIT',nodistribution} in context start_error

The example on the dockerhub page just show setting the hostname. But because this is ECS on an AWSVPC I can't just set hostname in the container definition. As per the manual:

Note

The hostname parameter is not supported if you are using the awsvpc network mode.

I only need to convince rabbitmq the hostname is static. I don't care what the domain or hostname is externally, just what the the process inside the container thinks is the hostname. Is there any other way to set this other than the hostname in the task definition?

This question feels too obvious to not already have an answer but all of my google searches are coming up with nothing.

In docker (docker-compose) you can easily get the IP of another container by hostname. If you have explicitly created and assigned a network this just works out of the box. So a container foo can connect to another container bar simply by looking up the hostname "bar".

I'm trying to achieve something similar in AWS ECS to allow a container from one service to talk to a container in another service. The ECS cluster has multiple hosts so I don't think I can just specify bridge type networking.

Note that the protocol I need this for is not HTTP so an application load balancer is already out of the question.

Is there a recommended way to let one container discover another on the same cluster via hostname?

I'm taking steps to harden my outgoing postfix SMTP server. I want to prevent users from spoofing their sender address.

When sending email with postfix SMTP the sender is identified in three ways:

1. In the email itself there is a From: header - as defined in RFC 822
2. When sending with SMTP there is a MAIL FROM address - as defined in RFC 821
3. When logging into postfix (SMTP) a client specifies a username which can also be interpreted as the sender

Looking in the manual, I believe that reject_sender_login_mismatch and reject_unlisted_sender will together guarantee that the MAIL FROM email as associated with the postifx login. That's 2 and 3 above match.

I'm stuck on how to ensure that all three match. I'm happy if the result just re-writes the message rather than rejecting but either would be fine. However I don't see a way to do this. I can see that cleanup will add a missing From: header but I see no way to check and rewrite one.

Thanks to those who pointed out another similar question. Unfortunately this has been of no help as the question has two answers, one is very vague the other appears to be wrong.

It is fairly standard to receive a significant number of minor hacking attempts each day trying common username / passwords for services like SSH and SMTP. I've always assumed these attempts are using the "small" address space of IPv4 to guess IP addresses. I notice that I get zero hacking attempts on IPv6 despite my domain having AAAA Name records mirroring every A Name record and all IPv4 services are also open to IPv6.

Assuming a public DNS (AWS route 53) with an obscure subdomain pointing to a reasonably randomised /64 suffix; Are IPv6 addresses and / subdomains remotely discoverable without trying every address in a /64 bit prefix or every subdomain in a very long list of common names?

I am of course aware that crawling the web looking for listed (sub)domain names is simple enough. I'm also aware that machines on the same subnet can use NDP. I'm more interested in whether DNS or the underlying protocols of IPv6 allow discovery / listing unknown domains and addresses by remote.

I'm looking for a way to take daily backups of an AWS bucket as incremental backups. These are to be stored offline and away from AWS.

For other storage systems (such as NAS drives) I use a daily rsync for backups. Using rsync's --link-dest switch, I'm able to take a full snapshot every day of the remote file system. Any files which have not changed since the previous backup are hardlinked to the previous backup. This means that full daily snapshots only take the storage space of incremental backups.

I would like to setup something similar for an amazon S3 bucket. There's 20GB in the bucket but only ~50MB changes per day.

Note this is backup the content of an S3 bucket, NOT backup other content to S3 bucket.

I can see how I would use the AWS CLI tools to do full backups. I don't see how I can do incremental backups.

I guess I could (daily) synchronise S3 to a local hard drive, then daily backup the local hard drive. This feels very clunky.

Edit

This was intended as a simple technical question, not a general discussion of backup security. But since I'm being asked "why do you need this", I now see I need to explain basic principles of backups.

Anecdote: I recently witnessed a third party IT provider drop (entirely) an S3 bucket because of a miscommunication. This could have been very costly (~£100K of recent work, ~£1M total work). Luckily we happened to also have copies on our local laptops and for only £1K we rebuilt the content for them.

It has renewed my conviction that the only valid "backup" is on an isolated system stored offsite and offline, and with a media rotation that effectively implements a time lock. Other backups can enhance, providing more rapid recovery etc... but holding all your AWS backups on your own AWS account just isn't safe because ... user error.

I'm trying to use quagga with RIP2 to route traffic through an openvpn server.

At the moment I'm having trouble because it's reporting the VPN subnet (172.19.2.x) as just a single IP (172.19.2.1). I've clearly missed a step but I'm not finding anything obvious in the documentation to point out why this is happening.

My Draytek (2820) router reports it's routing table as:

Key: C - connected, S - static, R - RIP, * - default, ~ - private
*            0.0.0.0/ 0.0.0.0          via 132.93.20.1      WAN2
C       132.93.20.0/ 255.255.254.0    directly connected    WAN2
C~      192.168.10.0/ 255.255.255.0    directly connected    LAN 
R~        172.19.2.1/ 255.255.255.255  via 192.168.10.10     LAN  (2/88050)

The zebra.conf configuration on the VPN server (192.168.10.10) is:

! -*- zebra -*-
hostname jupiter-zebra
password [removed]
enable password [removed]

! Tried adding this but didn't help
! ip route 192.168.10.0 255.255.255.0 eth0
! ip route 172.19.2.0   255.255.255.0 tun0

log file /var/log/quagga/zebra.log

The ripd.conf configuration on the VPN server is:

hostname jupiter-rip2
password [removed]

router rip
 network 192.168.10.0/24
 network 172.19.2.0/24
log file /var/log/quagga/ripd.log

Is there anything I've clearly missed to get my VPN server to report it's VPN subnet (172.19.2.x) instead of only it's VPN ip (172.19.2.1)?

I'm struggling to get apache2 to serve a file with the correct mime type. I'm using an installation of apache 2.2 installed with apt-get on Ubuntu.

The file in question is a maven repository file called maven-metadata.xml.md5.

For some reason apache is insisting this is of type Content-Type: application/xml This is clearly incorrect since the entire content of the file:

443219553065c4885947185d40d2a04e

I can only assume this decision is being made by apache because of "xml" in the file name, but it makes no sense in context.

I've tried adding md5 to /etc/mime.types as text/plain but I still get the same result.

The full headers are:

HTTP/1.1 200 OK
Date: Fri, 19 Dec 2014 22:58:30 GMT
Server: Apache/2
Last-Modified: Thu, 18 Dec 2014 14:56:57 GMT
ETag: "6e8e3cb-20-50a7ecdb68040"
Accept-Ranges: bytes
Content-Length: 32
Keep-Alive: timeout=30
Connection: Keep-Alive
Content-Type: application/xml

Any suggestions?

Edit I think I may have missed an obvious step like restarting the server after adding md5 to /etc/mime.types.

I'm setting up a LAN to LAN VPN using openvpn. For reasons of easy certificate management I wish to re-use the server certificates which already exist on each host. I've set in the config file remote-cert-tls server but this still errors:

Sun Aug 10 19:33:45 2014 176.126.242.99:37837 VERIFY ERROR: depth=0, error=unsupported certificate purpose: C=GB, ST=x, L=x, O=x x, OU=x, CN=x.x.x
Sun Aug 10 19:33:45 2014 176.126.242.99:37837 TLS_ERROR: BIO read tls_read_plaintext error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
Sun Aug 10 19:33:45 2014 176.126.242.99:37837 TLS Error: TLS object -> incoming plaintext read error
Sun Aug 10 19:33:45 2014 176.126.242.99:37837 TLS Error: TLS handshake failed

The client certificates are of the form:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 16 (0x10)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=y, ST=y, L=y, O=y y, OU=y, CN=y
        Validity
            Not Before: Aug  9 13:23:53 2014 GMT
            Not After : Aug  9 13:23:53 2015 GMT
        Subject: C=x, ST=x, L=x, O=x, OU=x, CN=x
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            Netscape Comment:
                Signed by y y
            X509v3 Subject Key Identifier:
                ...
            X509v3 Subject Alternative Name:
                DNS:x.x.x
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            X509v3 Authority Key Identifier:
                ...
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection

I'm note clear on exactly which feature of the certificate file is causing the error and what to change in the config file to fix it.

Edit

For additional Detail here's the Server and Client config

server 172.19.1.0 255.255.255.0
local 192.168.10.10
port 1195
proto udp
dev tun

ca /etc/ssl/certs/me.pem
cert /etc/ssl/certs/local/server.crt
key /etc/ssl/private/server.key
dh dh1024.pem

ifconfig-pool-persist ipp.txt

keepalive 60 720
comp-lzo

user nobody
group nogroup

persist-key
persist-tun

status openvpn-status.log
verb 3

and

client
dev tun
proto udp
remote x.x.x 1195
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun

ca /etc/ssl/certs/me.pem
cert /etc/ssl/certs/local/server.crt
key /etc/ssl/private/server.key

ns-cert-type server
comp-lzo
verb 3

I'm setting up a VPN network for multiple offices Using OpenVPN. This is dual purpose:

1. LAN to LAN VPN allowing traffic to be seamlessly routed between offices.
2. Allowing users to connect onto the network when they are out of the office.

This leads to the need for two separate authentication mechanisms:

1. Client Certificate for the LAN to LAN
2. Single Sign On User/Password (without client certificate) for the users.

According to the documentation:

• auth-user-pass-verify alone will require every user to provide Client certificate AND User / Password
• auth-user-pass-verify and client-cert-not-required will reqire only the User / Password

Question: How can I I configure Open VPN to Authenticate some clients with just the client certificate and other clients with Just the User / Password?

I'm working through a process of hardening my server security against the daily hacking attempts that arise the moment you attach a server to an IP and give it a domain name. I get anywhere from 1 to 8 brute force attempts daily to access SSH as either root or just brute force trying different names and this is on a server with no public reputation (its not running any big websites etc). Because of the way I have my SSH server configured I'm pretty sure these attempts will also fail, but I really dont like letting people try.

I have of course also set up connection rate limiting for the more sensitive services including SSH.

What I'm doing at the moment:

I can see from my auth.log that PAM does get the remote IP address of those trying to login and I'm currently using a script which periodically scans for these failed attempts and adds an IP block the fire wall.

What I'd like to do:

What I want to do is make this IP banning process much quicker to respond. That is rather than waiting for the polling script to pick it up, I want a PAM module to count successive failed attempts from an IP (not a service or user) and take some action either such as:

• refuse all future login attempts from that IP
• fire a command which will add a rule to the firewall to ban the IP completely

The Question:

Is there already a good PAM module which can take note of the IPs failing authentication or do I need to write my own?

I've recently been noticing some "odd" behavior from my ISP's DNS server when resolving non-existent domain. It's been causing me some problems and I was wondering if these DNS servers are actually conforming to the standard.

It started with them adding the feature that if you request an incorrect domain, it directs you to a search page that they run. I guess this is useful for some people when web browsing, but it borks things up for other applications particularly if the search server has some other open ports such as SMTP.

To get round this, I changed preferences with my ISP to turn this feature off. Now my ISP's DNS does something lot stranger. When a domain can not be found, it returns MY own IP address as the A name for that non-existent domain. So now if I click a link which no-longer exists, it bounces me to my own home page (run through the same Internet connection).

I'm trying to understand why being bounced to my own IP address would ever be useful. More to the point, is this the standard or are they breaking the standard for some purpose best known to themselves?

I'd like to create a custom program for handling incoming mail from postfix. I'd like to hook this into postfix by making it the mailbox_command. The manual indicates that there are a number of environment variables which are set when this command is invoked, however it doesn't say much else and doesn't include any particular requirements of this command.

Specifically I'd like to know:

1. how (if at all) the return code is used by postfix (what are the expected return codes and what effect do they have).
2. The mail itself appears to be handed to the command by passing it into the stdin. Is this a pipe or a temp file or is this not defined? The difference this makes is that a temp file can be memory mapped where as a pipe needs to be read byte by byte.
3. Can this command deffer accepting a mail (effectively tell postfix to try again later) - this could be answered by (1).
4. Is the stdout / std error logged or otherwise used in any way?

Does anyone know where this information can be found?

I'm in the process of configuring postfix on my server and I was wondering why people do not run the postfix queues from a ram disk?

The default home directory containing all the queues is (for the Ubuntu distribution) /var/spool/postfix. This of course is a folder which exists on hard disk under normal circumstances.

The basic answer which I've found from Google is that it protects against mail getting lost if the server crashes.

The question this still leaves me with is this: If the server did crash and the postfix queues were being stored on disk, would they not most likely be lost due to the kernel's internal hard disk caching. From what I understand, these files are pretty short lived. I'm struggling to see the difference between this and running more explicitly in RAM and saving to HD as postfix is shut down cleanly.

Have I missed something obvious here?

I need to re-build a cluster due to some incorrect parameters set when it was first created. In particular every database in the cluster is set to LC_COLLATE of en_GB.UTF8 and needs to be changed to C.

I'm comfortable with backup up the cluster with pg_dumpall, creating a new cluster with the correct configuration parameters and then restoring the backup to the new cluster.

To keep the risk as low as possible I would like to leave the existing cluster as unchanged as possible so that it can be started on request, but does NOT auto start on a call such as sudo service postgresql start. There is easily enough disk space to do this. Edit: To be clear on this I want to leave the old cluster installed, just not started.

How can I leave the cluster in place, but not have it started automatically along with the new (correctly configured) replacement? That is, how do I stop PostgreSQL from starting the old cluster, even though it is starts the new one?