halfer's questions

Introduction

I recently got a simple web app working on a three-node Ubuntu Server with MicroK8S. I decided to try rebuilding my cluster and reinstalling everything using YAML manifests, to ensure the process was replicable. However, the app is now not reachable from outside of the cluster. I am seeking debugging techniques to drill into why the NodePort is apparently not creating a TCP listener on all nodes.

Here are my nodes:

The cluster has again elected to run the workload on the third node, Yamazaki. I expect any web traffic hitting Arran or Nikka to be internally re-routed to Yamazaki to be serviced, as was happening previously.

What I did

From the previously working cluster/app, here is what I did to reset everything:

1. Do microk8s leave on all follower nodes

2. Do microk8s kubectl delete node <nodename> on the leader for each follower node (they were not removed automatically when they left)

3. Do microk8s reset on all nodes

4. Enable addons (dns, ingress). I don't know if either are necessary

5. Create join command on leader, microk8s add-node for each follower

6. Run a fresh join command microk8s join <ip>/<token> on each follower

7. Run microk8s status on any node to ensure cluster is in HA mode

8. Sideload an app image tarball from the leader, using microk8s images import workload.tar

9. Launch the app via microk8s kubectl apply -f k8s-manifests/production/pod.yaml -f k8s-manifests/production/nodeport.yaml

   Here is the Pod:

    apiVersion: v1
    kind: Pod
    metadata:
      name: k8s-workload
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"k8s-workload","namespace":"default"},"spec":{"containers":[{"image":"k8s-workload","imagePullPolicy":"Never","name":"k8s-workload","ports":[{"containerPort":9090,"protocol":"TCP"}]}]}}
    spec:
      containers:
      - image: k8s-workload
        imagePullPolicy: Never
        name: k8s-workload
        ports:
        - containerPort: 9090
          protocol: TCP
   

   Here is the NodePort:

    apiVersion: v1
    kind: Service
    metadata:
      name: np-service
    spec:
      type: NodePort
      ports:
        - port: 9090
          targetPort: 9090
          nodePort: 30090
      selector:
        run: k8s-workload
      # This should not be needed, but it didn't help
      # this time anyway
      externalIPs: [192.168.50.251]
   

10. Check the app is running via an internal container call, microk8s kubectl exec -ti k8s-workload -- curl http://localhost:9090 - this is fine

11. Check the app is running via a port forwarder on any node, microk8s kubectl port-forward pod/k8s-workload 9090 --address='0.0.0.0' - this is fine

12. Nodes not listening externally (curl http://localhost:30090 gets a refused connection, same with any node IP address from a non-cluster machine on the LAN)

System state

Here is what is running from microk8s kubectl get all -o wide:

NAME               READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
pod/k8s-workload   1/1     Running   0          20h   10.1.134.193   yamazaki   <none>           <none>

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE     SELECTOR
service/kubernetes   ClusterIP   10.152.183.1     <none>           443/TCP          35d     <none>
service/np-service   NodePort    10.152.183.175   192.168.50.251   9090:30090/TCP   3d21h   run=k8s-workload

I don't know what service/kubernetes is, I assume it is just part of the standard K8S infra.

Observations

I think this article is saying that my web app needs to be a service, but I only have a pod. I think that when this was working previously, I only had a pod, but the cluster had gotten into a bit of a mess, so it is possible that a service version of the app was running at the same time as the pod version.

The article also suggests that I ought to be using an ingress system. However, given that a NodePort is my present learning focus, I don't want to give up with it just yet. Ingress can come later.

I think I can be sure that there are no firewall issues, since any connections to port 30090 are rejected even in a console session on a node in the cluster.

I would like to run something like microk8s kubectl logs service np-service, to see what the NodePort is doing, but the logs subcommand only works on pods.

What can I try next?

I am learning K8S using MicroK8S. I have a three-node cluster, each node having 16G of RAM. The cluster has entered HA mode automatically. The cluster sits on my home LAN.

Here are my nodes:

Set-up

I have a web app running on a pod in the cluster. It responds on port 9090. Here is how I got it running.

I have a image on a development laptop that I turn into a tarball:

docker save k8s-workload > k8s-workload.docker.tar

I then send that tarball to the leader of the cluster:

scp k8s-workload.docker.tar 192.168.50.251:/home/myuser/

I then sideload this image into all nodes on the cluster:

root@arran:/home/myuser# microk8s images import < k8s-workload.docker.tar
Pushing OCI images to 192.168.50.251:25000
Pushing OCI images to 192.168.50.135:25000
Pushing OCI images to 192.168.50.74:25000

I then verify the MIME type and the checksum of the image, on every node, as I had some problems with that:

root@arran:/home/myuser# microk8s ctr images list | grep workload
docker.io/library/k8s-workload:latest   application/vnd.docker.distribution.manifest.v2+json    sha256:725b...582b 103.5 MiB linux/amd64

Finally I run the workload, ensuring that K8S does not try to pull an image (it is unnecessary, but the default policy is to try anyway):

root@arran:/home/myuser# microk8s kubectl run k8s-workload --image=k8s-workload --image-pull-policy='Never' --port=9090
pod/k8s-workload created

I then confirm that this was successful, from the leader node:

root@arran:/home/myuser# microk8s kubectl get pods -o wide
NAME           READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
k8s-workload   1/1     Running   0          83m   10.1.134.216   yamazaki   <none>           <none>

Running the app

In order to access the web app from my development laptop, I expose the app to start with on just one node. The pod is running on the Yamazaki node, so initially I run this from that node:

root@yamazaki:/home/myuser# microk8s kubectl port-forward pod/k8s-workload 9090 --address='0.0.0.0'
Forwarding from 0.0.0.0:9090 -> 9090

This works fine.

Problem

I would like to access the app by making a request to any node in the cluster, and not just this one. Currently the app only runs on one node and I would like it to work even if I make a web request to another node.

I know that K8S has the internal networking to do what I want. For example, if I run the port-forward command on Arran (and kill the same on Yamazaki) then the app will still work, even though the pod is running on Yamazaki only. But I can still only access the app from one IP (Arran, where the port forwarder is running).

Of course, I could do what I want by running the port-forwarder on an SSH session on every node. But I'd like to run something that survives after all SSH sessions are killed.

Ideally I would like to do this with a console command, but I wonder if I will need a YAML manifest for this. From my research so far, I think I need a ClusterIP.

Update/research 1

I'm reading the K8S manual page on bare-metal load balancing. Part of the page recommended that the user apply a 646-line config file, which would be pretty counterproductive for a learning scenario.

This sample seems more sensible, but it is not clear there how the LB is being instructed to run on all hosts.

Update/research 2

I also found this resource specifically for MicroK8S, which recommends an ingress addon. Unfortunately this requires my workload be set up as a Service, and for now I only have a Pod, so think this is out.

I continue down the frustratingly stop-start road of learning Kubernetes (specifically MicroK8S).

I build an image locally on a development laptop thus:

docker build -t k8s-workload .

This is a simple PHP web app that reports some request metadata. It builds successfully:

Sending build context to Docker daemon  13.82kB
Step 1/5 : FROM php:8.2-cli-alpine
 ---> c5f1f9770838
Step 2/5 : WORKDIR /root
 ---> Using cache
 ---> 492c997c963b
Step 3/5 : RUN apk update && apk upgrade
 ---> Using cache
 ---> f91505d5fe68
Step 4/5 : COPY src /root
 ---> 02bcc72dfc97
Step 5/5 : CMD ["sh", "/root/bin/start-server.sh"]
 ---> Running in 6bc3b72365e4
Removing intermediate container 6bc3b72365e4
 ---> 0c8a405b06af
Successfully built 0c8a405b06af
Successfully tagged k8s-workload:latest

I create a tarball from this so it can be sent to my three-node cluster:

docker save k8s-workload > k8s-workload.docker.tar

I then send it to the leader in the cluster (though I assume it could be sent to any of them):

scp k8s-workload.docker.tar 192.168.50.251:/home/myuser/

This is all looking good so far. Now I want to sideload the image into all nodes in the cluster:

root@arran:/home/myuser# microk8s images import < k8s-workload.docker.tar
Pushing OCI images to 192.168.50.251:25000
Pushing OCI images to 192.168.50.135:25000
Pushing OCI images to 192.168.50.74:25000

With that looking successful, I tried to create a workload:

root@arran:/home/myuser# microk8s kubectl create deployment k8s-workload --image=k8s-workload

Finally let's get the status of this pod:

root@arran:/home/myuser# microk8s kubectl get pods
NAME                            READY   STATUS             RESTARTS   AGE
k8s-workload-6cdfbb6b59-zvgrl   0/1     ImagePullBackOff   0          35m

OK, that doesn't look good. There was also an error of ErrImagePull, but that seems to have been replaced now.

How can I debug why an image won't start?

I discovered a way to list the images on a node. I have found my newly built image on the leader node:

root@arran:/home/myuser# microk8s ctr images list | grep workload
docker.io/library/k8s-workload:latest   application/vnd.docker.distribution.manifest.v2+json    sha256:725b...582b 103.5 MiB linux/amd64

So the image is available. I can get some logs on the problem, but it does not reveal anything that I don't already know:

root@arran:/home/myuser# microk8s kubectl logs k8s-workload-1cdfaa6c49-zvgrl
Error from server (BadRequest): container "k8s-workload" in pod "k8s-workload-1cdfaa6c49-zvgrl" is waiting to start: trying and failing to pull image

What can I try next? No node would actually need to pull an image, as far as I know, since they are all available on each node.

Update 1

I was hesitant to add too many problems to the one question, but on balance I think they are worth adding, since they are all obstacles to obtaining one result: successfully deploying a trivial workload on K8S.

When describing the single pod within the single deployment, I noticed it showed me this error:

kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.

Yikes! Another thing that doesn't work out of the box. I have fixed this the MicroK8S way using this answer. It hasn't solved the problem, but at least I am knocking the roadblocks on the head, one by one.

Update 2

I wanted to check that the side-loaded image was valid, so I did this on the leader:

root@arran:/home/myuser# docker load < k8s-workload.docker.tar

That unpacks fine:

bb01bd7e32b5: Loading layer [==================================================>]  7.618MB/7.618MB
e759f13eb8bc: Loading layer [==================================================>]  6.015MB/6.015MB
1a72c946ba2b: Loading layer [==================================================>]  12.29kB/12.29kB
9bbacedbd5e4: Loading layer [==================================================>]  6.144kB/6.144kB
53b5e1394bc2: Loading layer [==================================================>]  12.08MB/12.08MB
aff825926dad: Loading layer [==================================================>]  4.096kB/4.096kB
c76bce6229c6: Loading layer [==================================================>]   71.7MB/71.7MB
0503c7346508: Loading layer [==================================================>]   12.8kB/12.8kB
8c2f9e7d94bb: Loading layer [==================================================>]  65.54kB/65.54kB
7e0ad9ed4982: Loading layer [==================================================>]  10.97MB/10.97MB
b99f234d8751: Loading layer [==================================================>]  5.632kB/5.632kB
Loaded image: k8s-workload:latest

I then run it on the leader on a custom port (i.e. this is in Docker, not K8S):

root@arran:/home/myuser# docker run -p 9000:80 -it k8s-workload

This responds via cURL as I would expect from another machine on the LAN.

Update 3

It occurred to me that the "namespaced" image name might a different - should I be specifying docker.io/library/k8s-workload:latest rather than k8s-workload? I tried both, and I find I get the same result.

So here is the latest error:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m46s                  default-scheduler  Successfully assigned default/k8s-workload-68c899df98-qhmhr to yamazaki
  Normal   Pulling    3m17s (x4 over 4m45s)  kubelet            Pulling image "k8s-workload"
  Warning  Failed     3m15s (x4 over 4m43s)  kubelet            Failed to pull image "k8s-workload": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/k8s-workload:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:e823...45c8: not found
  Warning  Failed     3m15s (x4 over 4m43s)  kubelet            Error: ErrImagePull
  Warning  Failed     2m52s (x6 over 4m43s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m37s (x7 over 4m43s)  kubelet            Back-off pulling image "k8s-workload"

OK, so I now have a bit more detail. What does the "failed to unpack image" error actually mean?

Update 4

A helpful answer below suggests that I might need to set a pull policy to get K8S to expect the image to be available on each node, and that it should not try to pull them (it does not exist anywhere remotely).

However when taking the supplied advice, although I get a different error code (CreateContainerError), the underlying cause is the same:

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  64s               default-scheduler  Successfully assigned default/k8s-workload to yamazaki
  Normal   Pulled     6s (x7 over 62s)  kubelet            Container image "k8s-workload" already present on machine
  Warning  Failed     6s (x7 over 62s)  kubelet            Error: failed to create containerd container: error unpacking image: unexpected media type text/html for sha256:1f2c...753e1: not found

Update 5

I have reported this as a bug for now, though I would still welcome answers here.

Update 6

On the basis that dogged levels of persistence is mysteriously good for the soul, I tried deleting the image using the ctr subcommand. This is on a follower node:

root@yamazaki:/home/myuser# microk8s ctr images rm docker.io/library/k8s-workload:latest
docker.io/library/k8s-workload:latest

Then using the same subcommand I reimported:

root@yamazaki:/home/myuser# microk8s ctr images import k8s-workload.docker.tar 
unpacking docker.io/library/k8s-workload:latest (sha256:725b...582b)...done

Since this operates at the node level and not the cluster, I did this for each of the three nodes.

I then used the run command, since this permits a pull policy to be set, and I don't want to conflate the unpack problem with a pull problem on top. This is back on the cluster leader:

root@arran:/home/myuser# microk8s kubectl run k8s-workload --image=k8s-workload --image-pull-policy='Never' --port=80
pod/k8s-workload created

I then describe the resulting pod, and get a familiar error:

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  36s               default-scheduler  Successfully assigned default/k8s-workload to yamazaki
  Normal   Pulled     6s (x5 over 35s)  kubelet            Container image "k8s-workload" already present on machine
  Warning  Failed     6s (x5 over 35s)  kubelet            Error: failed to create containerd container: error unpacking image: unexpected media type text/html for sha256:5f76...a3aa: not found

Paradoxically this is reassuring - sending an image to each node individually is a right hassle, and thus I'd want the cluster level image import to work. I suspect it will once I have got to the bottom of the unpack problem.

Update 7

Right, I have spotted something. The image tarball on all nodes has an identical checksum, as one would expect. But when it is imported, one node reports the wrong type against the image. These are lightly reformatted for ease of comparison:

Node "Arran":

docker.io/library/k8s-workload:latest  
application/vnd.docker.distribution.manifest.v2+json  
sha256:725b...582b 103.5 MiB  
linux/amd64  
io.cri-containerd.image=managed

Node "Yamazaki":

docker.io/library/k8s-workload:latest  
text/html  
sha256:5f76...a3aa 218.4 KiB  
-  
io.cri-containerd.image=managed

Node "Nikka":

docker.io/library/k8s-workload:latest  
application/vnd.docker.distribution.manifest.v2+json  
sha256:725b...582b 103.5 MiB  
linux/amd64  
io.cri-containerd.image=managed

It looks like the workloads consistently were elected to run on Yamazaki, and that's the node with the corrupted image. Now to reimport the image and get it to match the others...

I have built a little three-node Kubernetes cluster at home, for learning purposes. Each 16GB node runs Ubuntu Server and MicroK8S. I have set up a leader (arran) and two followers (nikka and yamazaki).

root@arran:/home/me# microk8s kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
arran      Ready    <none>   5d3h    v1.26.4   192.168.50.251   <none>        Ubuntu 22.04.2 LTS   5.15.0-71-generic   containerd://1.6.15
nikka      Ready    <none>   4d14h   v1.26.4   192.168.50.74    <none>        Ubuntu 22.04.2 LTS   5.15.0-71-generic   containerd://1.6.15
yamazaki   Ready    <none>   3d16h   v1.26.4   192.168.50.135   <none>        Ubuntu 22.04.2 LTS   5.15.0-71-generic   containerd://1.6.15

Here is the status of the cluster, with ingress and dashboard manually enabled. You can see it has switched into HA mode:

root@arran:/home/me# microk8s status
microk8s is running
high-availability: yes
  datastore master nodes: 192.168.50.251:19001 192.168.50.74:19001 192.168.50.135:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard            # (core) The Kubernetes dashboard
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated
  disabled:
    cert-manager         # (core) Cloud native certificate management
    community            # (core) The community addons repository
    dns                  # (core) CoreDNS
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation

Here are my running pods, which come from my manifest (see later):

root@arran:/home/me# microk8s kubectl get pods -o wide
NAME                               READY   STATUS    RESTARTS      AGE    IP             NODE       NOMINATED NODE   READINESS GATES
hello-world-app                    1/1     Running   1 (14h ago)   47h    10.1.134.199   yamazaki   <none>           <none>
my-pod                             1/1     Running   2 (14h ago)   5d1h   10.1.150.208   arran      <none>           <none>

Here are the services at present:

root@arran:/home/me# microk8s kubectl get services -o wide
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE     SELECTOR
kubernetes            ClusterIP   10.152.183.1     <none>        443/TCP         5d3h    <none>
nginx-service         NodePort    10.152.183.120   <none>        80:30000/TCP    2d12h   app.kubernetes.io/name=hello-world-app
hello-world-service   NodePort    10.152.183.205   <none>        80:30032/TCP    47h     app.kubernetes.io/name=hello-world-app
dashboard-service     NodePort    10.152.183.237   <none>        443:32589/TCP   47h     app.kubernetes.io/name=kubernetes

I suspect the problem is in the manifest, which I have built in a bit of a copy-and-paste fashion from the K8S and MicroK8s manuals:

apiVersion: v1
kind: Pod
metadata:
  name: hello-world-app
  labels:
    app.kubernetes.io/name: hello-world-app
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - name: http
          containerPort: 80
          protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  name: hello-world-service
spec:
  selector:
    app.kubernetes.io/name: hello-world-app
  ports:
  - port: 80
    targetPort: 80
  type: NodePort

---

# Not sure this will work - do we need a NodePort to the dashboard?
apiVersion: v1
kind: Service
metadata:
  name: dashboard-service
spec:
  selector:
    app.kubernetes.io/name: kubernetes
  ports:
  - port: 443
    targetPort: 443
  type: NodePort

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: http-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes
            port:
              number: 443

Now, I have a "hello world" app, I've given it a node port, and then exposed that using an ingress plugin. This has come available on http://192.168.50.251/ (port 80). However I have tried to do the same for the Kubernetes dashboard, by adding a port and an ingress route (port 443), but https://192.168.50.251/ points to "hello world" and not the dashboard as I intend.

The single-file manifest file has been fully applied with microk8s kubectl apply -f manifest.yml.

What can I try next?

Introduction

I have some TypeScript code that uses CDK to create an API Gateway and a Lambda. It works and deploys to a standard AWS URL. So far so good.

I now need to transfer the API Gateway so that it operates on a custom domain, so that it can set a cookie in a web app. This is proving far harder, and I suspect I am having difficulty because I am new to TypeScript, AWS, and CDK all at the same time. There are a number of documentation resources on the web, but most would require me to rewrite the precious little working code I have, which I am reluctant to do.

I have created a certificate manually, because that requires validation and thus it does not make sense to create it in code. Other than that I want all other resources to be created by CDK code in a Stack. In my view, it defeats the purpose of CDK if I have to configure things manually.

Problem

The below code deploys everything I need to gatekeeper.d.aws.example.com - a HostedZone, an ARecord, a LambdaRestApi and a Function (lambda). However it does not work because the NS records newly assigned to gatekeeper.d.aws.example.com do not match the ones in the parent d.aws.example.com.

I think this means that although d.aws.example.com is "known", the gateway subdomain cannot delegate to it.

Here is my working code:

// Create the lambda resource
const referrerLambda = new lambda.Function(this, 'EisReferrerLambda', {
    runtime: lambda.Runtime.NODEJS_14_X,
    handler: 'index.handler',
    code: lambda.Code.fromAsset(path.join(__dirname, '../../src/lambda')),
    environment: env
});

// Set up the domain name on which the API should appear
const domainName = 'gatekeeper.d.aws.example.com';

// TODO need to fetch it with an env var? Or read from environment?
const certificateArn = 'arn:aws:acm:us-east-1:xxx:certificate/yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy';

const certificate = acm.Certificate.fromCertificateArn(this, 'SslCertificate', certificateArn);

const hostedZone = new route53.HostedZone(this, 'EisReferrerHostedZone', {
    zoneName: domainName
});

// Add an A record
new route53.ARecord(this, 'DnsRecord', {
    zone: hostedZone,
    target: route53.RecordTarget.fromAlias(new targets.ApiGateway(apiGateway)),
});

// I think I need a DomainNameOptions object
const dno : DomainNameOptions = { certificate, domainName };

// Create the APIG resource
// See https://intro-to-cdk.workshop.aws/the-workshop/4-create-apigateway.html
const apiGateway = new apigw.LambdaRestApi(this, "EisReferrerApi", {
    handler: referrerLambda,
    // proxy = on means that the lambda handles all requests to the APIG,
    // instead of just explicit resource endpoints
    proxy: false,
    // deploy = on means that we get a default stage of "prod", I don't want
    // that - I'm creating a custom Deployment anyway
    deploy: false,
    // Point to a domain name options object
    domainName: dno
});

// Create an endpoint in the APIG
// https://docs.aws.amazon.com/cdk/api/latest/docs/aws-apigateway-readme.html#defining-apis
const items = apiGateway.root.addResource('gatekeeper');
items.addMethod('GET');  // GET /default/gatekeeper

// The deployment resource is just needed by the Stage system
const deployment = new apigw.Deployment(
    this,
    'EisReferrerDeployment',
    { api: apiGateway }
);

// Create a Stage (this affects the first component in the path
const stageName = 'default';
apiGateway.deploymentStage = new apigw.Stage(
    this,
    stageName,
    { deployment, stageName }
);

Question

As you can see from the code, I've found how to create an A record, but creating/modifying NS records seems harder. For a start, there does not seem to be an NSRecord class, at least based on exploring the class structure from my IDE autocomplete.

A rudimentary solution would allow me to create NS records with the fixed values that are set up elsewhere (in the AWS account that "owns" the domain). A better solution would be to read what those records are, and then use them.

Update

To see if my thinking is on the right track, I have run this deployment code, and manually modified the automatically assigned NS records in the HostedZone to match the records in the parent (in the other account). I think I have to wait for this change to seep into the DNS system, and I will update with the result.

Update 2

My manual adjustment did not work. I have therefore found a new thing to try (see "To add a NS record to a HostedZone in different account"):

// Commented out from earlier code
// const hostedZone = new route53.HostedZone(this, 'EisReferrerHostedZone', {
//     zoneName: domainName
// });

// In the account containing the HostedZone
const parentZone = new route53.PublicHostedZone(this, 'HostedZone', {
    zoneName: 'd.aws.example.com',
    crossAccountZoneDelegationPrincipal: new iam.AccountPrincipal('12345678012')
});

// In this account
const subZone = new route53.PublicHostedZone(this, 'SubZone', {
    zoneName: domainName
});

new route53.CrossAccountZoneDelegationRecord(this, 'delegate', {
    delegatedZone: subZone,
    parentHostedZoneId: parentZone.hostedZoneId,
    delegationRole: parentZone.crossAccountDelegationRole
});

This sounds exactly what I need, but I fear the AWS documentation is out of date here - crossAccountDelegationRole is rendered in red in my IDE, and it crashes due to being undefined when cdk diff is run.

Update 3

I am assuming the property mentioned above is a typo or a reference to an outdated version of the library. I am now doing this:

new route53.CrossAccountZoneDelegationRecord(this, 'delegate', {
    delegatedZone: subZone,
    parentHostedZoneId: parentZone.hostedZoneId,
    delegationRole: parentZone.crossAccountZoneDelegationRole
});

This feel tantalisingly close, but it crashes:

Failed to create resource. AccessDenied: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/CustomCrossAccountZoneDelegationC-xxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::yyyyyyyyyyyy:role/HostedZoneCrossAccountZoneDelegat-yyy

I wonder if I need to declare the IAM creds for the other account? I do have them.

I am not sure why permissions are needed, anyway - could it not just read the NS records in the other account and copy them to the local account? The DNS in the other account is public anyway.

I am willing to research fixing the IAM error, but this doesn't half feel like shooting in the dark. I might spend another two hours inching towards solving that sub-problem, only to find that the whole thing will fail for another reason.

I am looking to learn Kubernetes, specifically from this resource. I only want to run Docker containers, rather than full-fat VMs. It says that I need any of these as a base from which to work:

• Docker Desktop
• kind
• Minikube

I have plumped for Minikube, as I'm on Linux, and it's pretty easy to install. I'm using these instructions. The second doc link says:

To check if virtualization is supported on Linux, run the following command and verify that the output is non-empty:

grep -E --color 'vmx|svm' /proc/cpuinfo

This does not produce any output for me, since my standard development environment runs inside a VirtualBox container. I don't plan to change this - Docker, Docker Compose and Docker Swarm run very well here.

It later says:

Install a Hypervisor

If you do not already have a hypervisor installed, install one of these now:

• KVM, which also uses QEMU

• VirtualBox

This makes it seem that hypervisor CPU support is necessary, but I do not see why that would be.

However it does go onto say:

Minikube also supports a --driver=none option that runs the Kubernetes components on the host and not in a VM. Using this driver requires Docker and a Linux environment but not a hypervisor.

This appears to be something of an addendum, and is given much less prominence that the suggestion that virtualisation instructions are a requirement. Were the documentation authors just "covering all bases", or will I bump into a roadblock later on by continuing despite not having the required CPU?

I've done a little Docker Swarm in the past, and it was pretty uncomplicated - create a Swarm Manager on one node, create Swarm Workers on two more nodes, stick with a single Manager. I would like to learn more about Swarm, so I am following along the video series "Docker Swarm: Native Docker Clustering" by Nigel Poulton on Pluralsight.

However, these videos are a great deal more complicated than my Swarm experiences two years ago, possibly because they run on an old ("Standalone") version of Swarm. I would like to ask a clarifying question to see if the material I am now learning is rather too outdated to be worthwhile. (I will describe below what is on the video course, so that one does not need to be familiar with it to answer the question).

The video series, now four years old at this point, shows how to:

• Install a high-availability Discovery Service (Consul, Etcd and ZooKeeper are supported, and the author chooses Consul). It uses the progrium/consul image, which as of today has not been updated in five years
• Set up three Swarm Manager nodes on the Consul servers
• Set up three Swarm Worker nodes, on which Consul clients are also installed

The author acknowledges that this is all rather involved, and hints that, at the time of shooting the videos, core Docker might simplify this stuff in the future. He says (Building Your Swarm Cluster -> Installing an HA Discovery Service -> 5:39):

I would even put a wild punt out there, that at some point, Swarm might even not need us to do any of this discovery backend hand-crafting. Just because it's a bit of a pain, and if I know one thing about the folks at Docker, it's that they're all hell-bent on giving us this smooth, effortless, Docker experience, which this part kind of isn't at the moment.

And that feature appears to be what actually happened, if I am reading things right. From the manual:

You are viewing docs for legacy standalone Swarm. These topics describe standalone Docker Swarm. In Docker 1.12 and higher, Swarm mode is integrated with Docker Engine. Most users should use integrated Swarm mode.

Admittedly my previous experiences with Swarm were only using a single Manager, but from what I understand, one can add a Manager to an existing raft merely using docker swarm join-token.

To my question, then: this feature evolution of Docker Swarm makes me wonder if Consul offers any value for managing the Swarm itself. Can I do it? Should I? Is there more features that it offers over the integrated raft consensus system? Is this learning material now harmful?

(Aside: I've watched a couple of videos from this author, and they are excellent - if this one is outdated then that is outside of his control. Should it be the case that the material needs reshooting, it is the course/platform owners that would need to arrange that).

I have a single small VPS that has a variety of services on it, such as a WordPress installation and a few web apps. For some while I have run all the services on it as Docker containers. Since I have a variety of domains and subdomains pointing to this box, I use the frontend proxy Traefik to capture the web ports and then route them internally in a Docker network.

I start up Traefik like so:

#!/bin/bash

# Removes the restart policy from previous containers
CONTAINER_LABEL=traefik-instance
../../bin/remove-restart.sh $CONTAINER_LABEL

mkdir --parents /var/log/traefik
mkdir --parents /etc/letsencrypt-traefik

docker run \
    --label $CONTAINER_LABEL \
    --publish 80:80 \
    --publish 443:443 \
    --volume $PWD/traefik.toml:/etc/traefik/traefik.toml \
    --volume $PWD/rules:/etc/traefik/rules \
    --volume /etc/letsencrypt-traefik:/etc/letsencrypt-traefik \
    --volume /var/log/traefik:/log \
    --network dockernet \
    --detach \
    --restart always \
    traefik:1.6

This all works very nicely. I have recently discovered Docker Swarm, and would like to convert all of my containers to services, which will give me replication services, rolling updates and zero-downtime deployments. However, I would like to do the change piecemeal, so that Traefik can route to both Swarm services and ordinary (non-Swarm) containers.

So, to launch Traefik as a service, I am now doing the following. You'll notice I am using non-standard ports for the purposes of testing:

#!/bin/bash

# Using "traefik2" while I am experimenting with multiple services
mkdir --parents /var/log/traefik2
mkdir --parents /etc/letsencrypt-traefik

docker service create \
    --publish 8080:80 \
    --publish 8443:443 \
    --mount type=bind,source=$PWD/traefik.toml,target=/etc/traefik/traefik.toml \
    --mount type=bind,source=$PWD/rules,target=/etc/traefik/rules \
    --mount type=bind,source=/etc/letsencrypt-traefik,target=/etc/letsencrypt-traefik \
    --mount type=bind,source=/var/log/traefik2,target=/log \
    --network traefiknet \
    traefik:1.6

This works also, when pointed to a Swarm web service that appears on the same network.

So, I have two Docker networks (amongst the various defaults that Docker creates for itself) like so:

root@box:~/docker# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
1aa479f13faa        dockernet           bridge              local
k71hpg1n0lo9        traefiknet          overlay             swarm

This results in my having a working Traefik container that can see Docker containers, and a working Traefik service that can see Swarm services. However, they cannot see each other.

To try to fix this, I have tried to add the Docker network to the start-up of the Traefik Swarm service:

--network dockernet \

In other words, I want this service to connect to both the bridge (old) and the overlay (new) networks. Unfortunately I get this:

Error response from daemon: The network dockernet cannot be used with services. Only networks scoped to the swarm can be used, such as those created with the overlay driver.

Is there a way my new service can connect to the old network, or indeed is there a way my old containers can connect to the new network? I have tried searching for the error, but there do not seem to be many mentions of it at all; I wonder if this edge-case usage of Swarm has not been encountered by many folks yet.

(Of course, one solution is for me to convert all of my containers to services, but to avoid a big-bang change, I'd rather do it slowly if possible).

Trying attachable networks

I then deleted my services and tried this:

docker network rm traefiknet
docker network create driver=overlay --attachable traefiknet

I then recreated the Traefik service, and it starts up. It is evidently still working because it routes traffic to a service that has also joined the traefiknet overlay.

However, I have created a non-service container, and connected that exclusively to traefiknet, and the --network-alias I create with that cannot be seen by the service. Oddly, if I shell into this non-Swarm container it can ping the Swarm Traefik container, so the network works. (I have tried creating an Alpine shell service, connected to traefiknet, and from here I cannot ping either the container name of my non-Swarm container, nor its --network-alias).

Upgrading Docker

I have tried to upgrade Docker from 17.03.2-ce to 18.06.1-ce, because a phrase in the manual indicated that my old Docker version might be the cause of the problem:

Communicate between a container and a swarm service sets up communication between a standalone container and a swarm service, using an attachable overlay network. This is supported in Docker 17.06 and higher.

However, this also has not helped.

Background

I have four Windows servers, running Server 2012 R2, running Apache 2.4 on each. These are arranged into two pairs of load-balanced web servers, with the first pair facing the internet, and the second pair inside the LAN. There is an incoming firewall rule to allow port 80 traffic from the first to the second for API requests.

I am also running the Tableau Server visualisation software in the LAN, and my current task is to proxy traffic through the two sets of web servers to Tableau. I am using these rules on the outer web server:

/tableau-proxy/* : proxy to the inner web server
/*               : run a web app that carries Tableau content

On the inner web server, I am using these rules:

/tableau-proxy/* : proxy to Tableau and strip off the /tableau-proxy prefix
/*               : run an API app

For both sets of rules, I am using mod_proxy, mod_proxy_http, and mod_rewrite. The proxying itself is working; for example, a image request can be carried all the way from the internet, bounce off the two web servers, land on Tableau, and then be carried back to the user as PNG data.

Content rewriting problem

The last 5% of this problem is to rewrite HTML links coming back from Tableau so that the proxy subdir prefix is restored. This is on the LAN-side/inner web servers. The rule I am using is this:

/ -> /tableau-proxy

I am trying to use mod_proxy_html for this, but with the configuration I am using, I am getting a blank page. Interestingly, it is served with a response code of 200, so it is a bit harder to debug.

Configuration

The config I am using is this:

LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule xml2enc_module modules/mod_xml2enc.so
Include conf/extra/proxy-html.conf

<Directory "C:\Apache24\htdocs\public\tableau-proxy">
    LogLevel alert rewrite:trace2 proxy_html:trace2

    # Proxy requests to the Tableau LB
    RewriteEngine on

    # Here is a test Tableau server
    RewriteRule (.*) http://tabtest/$1 [P]

    # Don't make the x-forwarded-for header a list of proxy hops!
    # That will break redirects in Tableau
    ProxyAddHeaders Off

    ProxyHTMLExtended Off
    xml2EncDefault utf-8
    LogLevel debug

    ProxyHTMLEnable On
    ProxyHTMLURLMap / /tableau-proxy
</Directory>

I have also tried preserving the charset of the document as it comes back down the proxy path:

ProxyHTMLCharsetOut *

That does not seem to make a difference, and results in:

Info: Got charset utf-8 from HTTP headers

Error: Invalid argument: [client: IP] AH01427: xml2enc: Charset utf-8 not supported

(What, it doesn't support UTF-8?)

I have tried experimenting with different HTML doctypes, again with no change:

ProxyHTMLDoctype XHTML Legacy

I've tried adding the proxy HTML module as a filter, in case this does something more than ProxyHTMLEnable On, again to no avail:

SetOutputFilter proxy-html

I have seen some sporadic reports around the web of other folks experiencing this issue, and some of the extra items above represent my trying some suggested solutions, but the blank page persists. What can I try next?

I am currently working on a project for which the Apache web server is preferred, but Windows is mandatory. I have not used WAMP for some years, but I believe these days that it works well and is stable. We have four web servers, each in two load-balanced pairs. Apache is being run as a Windows Service.

A colleague has found some information that says it is safer to run Apache under a custom account rather than the "Local System Account". So, a local account has been created on each box and the credentials entered into the "Log On" tab in the Service Properties dialogue box.

From time to time when the Apache configuration changes, I need to restart each service. To do that I select the Apache2.4 entry in the Services window and click on the restart button. However, for all boxes, it is touch-and-go as to whether it will be successful. When it fails, I get a Windows error that says:

Windows could not start the Apache2.4 service on Local Computer.

Error 1069: The service did not start due to a logon failure.

So, I dutifully add the service password in again, whenever this happens, and then it always starts up. One guess I have is that the dialogue is forgetting the password it has been given. It does not ever forget the username.

I have examined the permissions given to this local user in Local Security Policy -> Local Policies -> User Rights Assignment -> Log on as a service Properties, and found that the user is in there normally, but if it becomes missing, then an Apache restart will fail.

When researching this, I found this identical scenario (although the service in that case is PostgreSQL). However, the accepted answer of using the Local System Account is not an answer to the question posed, in my view. The question ought to be: why does the user lose the right to log on as a service?

I am building upon gists/pure-ftpd to create FTP server instances in a Docker container. Here is my server start command, which I am hacking on:

# Args explained at http://debianhelp.co.uk/pureftp.htm
#
# -P -Force the specified IP address in reply to a PASV/EPSV/SPSV command
# -S -Connections are accepted on the specified IP and port
# -l -This is the authentication type, in the form "protocol:path"
# -E -Only allow authenticated users (if you wanted anonymous only you would substitute -e).
# -j -If the user doesn't have a home directory create it at first login.
# -R -Disallow the usage of the chmod command.
# -B -Instruct the standalone server to start in the background
# -g -Custom pidfile location (defaults to /var/run/pure-ftpd.pid)
# -d Verbose logging
/usr/sbin/pure-ftpd \
    -S $PUBLIC_HOST,21 \
    -P $PUBLIC_HOST \
    -p $MIN_PASV_PORT:$MAX_PASV_PORT \
    -g $PID_FILE \
    -l unix \
    -d \
    -E \
    -j \
    -R \
    -B

I am using the unix permission system, since I don't need the separate user database system. However, I am creating users and they don't seem to work, so I want to see some logs from PureFTPd, and the server just does not want to do any logging for me.

I have tried the user account via SSH on the same box, and this works, so I know the username and password is valid.

When playing around with this, I have removed the -B background switch in PureFTPd, but I see no logs on stdout. Much of the documentation/help around the internet says that logs are sent to syslog, but that's not running in this container, since it is a pretty bare Alpine build. I tried adding Rsyslog, while being conscious that "side projects" are a time-sink, and I got this:

rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog.so failed [v8.31.0 try http://www.rsyslog.com/e/2145 ]

So I have given up with that, and tried --with-altlog /var/log/ftp.log, and that hasn't written anything either.

I could switch to Ubuntu where syslog (etc) is probably available and running by default, but I don't want to do that only to find that logging is broken in PureFTPd anyway. Is there a simple fix?

Addendum

While it seems pointless (for my use case) to duplicate the Unix user system with a virtual FTP user database, I have tried this, and it works. Instead of -l unix, I use -l puredb:/etc/pureftpd/pureftpd.pdb. This gets my FTP server working, at least.

Update 2

I have fixed the Unix users problem. I believe that it was due to how I was creating users - they did not have a shell or a group. I have resolved that, and this mode of authentication now seems to work.

Answers on the logging aspect are still sought.

I am building a Dockerised system that will create and tear down various server types in an HTTP API, for the purposes of integration testing. I am starting the project by experimenting how to create and run multiple FTP servers simultaneously.

I want to run FTP servers on separate IP addresses, so that tests can create and tear down servers without interfering with each other. In future, that should allow tests to be parallelised if necessary.

In my Docker container, I have set up various networks, as per this Docker Compose config:

version: '2'
services:

  missive-box-of-tricks:
    image: missive-box-of-tricks
    networks:
      network-01:
        aliases:
        - network-01
      network-02:
        aliases:
        - network-02
      network-03:
        aliases:
        - network-03
      network-04:
        aliases:
        - network-04
      network-05:
        aliases:
        - network-05
      (trimmed - there's 10 of these)

networks:
  network-01:
    driver: bridge
  network-02:
    driver: bridge
  network-03:
    driver: bridge
  network-04:
    driver: bridge
  network-05:
    driver: bridge
  (trimmed - there's 10 of these)

This results in a set of networks inside the container (trimmed again):

~ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:14:00:02  
          inet addr:172.20.0.2  Bcast:172.20.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3249 (3.1 KiB)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:AC:16:00:02  
          inet addr:172.22.0.2  Bcast:172.22.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3249 (3.1 KiB)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 02:42:AC:15:00:02  
          inet addr:172.21.0.2  Bcast:172.21.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3249 (3.1 KiB)  TX bytes:0 (0.0 B)

eth3      Link encap:Ethernet  HWaddr 02:42:AC:17:00:02  
          inet addr:172.23.0.2  Bcast:172.23.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3249 (3.1 KiB)  TX bytes:0 (0.0 B)

eth4      Link encap:Ethernet  HWaddr 02:42:AC:18:00:02  
          inet addr:172.24.0.2  Bcast:172.24.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:27 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3249 (3.1 KiB)  TX bytes:0 (0.0 B)

The network aliases are resolvable, so I'd expect for the FTP server to be able to listen to network-01 etc. I use this command to start up a server, from inside the container:

PUBLIC_HOST=network-01

# -P -Force the specified IP address in reply to a PASV/EPSV/SPSV command
# -S -Connections are accepted on the specified IP and port
# -l -This is the authentication type, in the form "protocol:path"
# -E -Only allow authenticated users (if you wanted anonymous only you would substitute -e).
# -j -If the user doesn't have a home directory create it at first login.
# -R -Disallow the usage of the chmod command.
# -B -Instruct the standalone server to start in the background
# -g -Custom pidfile location (defaults to /var/run/pure-ftpd.pid)
# -d Verbose logging

/usr/sbin/pure-ftpd \
    -S $PUBLIC_HOST \
    -p $MIN_PASV_PORT:$MAX_PASV_PORT \
    -g $PID_FILE \
    -l $USER_DB \
    -d \
    -E \
    -j \
    -R

I have reset the PID and user database files, so as to avoid conflict, but in fact even a single server won't start. I get this error:

Unable to start a standalone server: Address not available

I get the same if I add a PASV declaration:

-P $PUBLIC_HOST \

I also get the same if I put the IP address in explicitly, rather than using an environment variable.

So, why is this IP address not available? Is there another way I can create a range of IP addresses inside a Docker environment and then bind a PureFTP instance to listen to just one?

I have a general purpose VPS, and am Dockerising the apps on it. There will be around 5-6 containers on it, and very little else, so the box can be trivially rebuilt as required.

For each app I have a start script. A WordPress container looks like this:

#!/bin/bash

# Get the host IP address
export DOCKER_HOSTIP=`ifconfig docker0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1`
echo "Connecting to database on Docker host ${DOCKER_HOSTIP}"

docker run \
    --add-host=docker:${DOCKER_HOSTIP} \
    --network dockernet \
    --network-alias jonblog \
    --detach \
    --restart always \
    --rm \
    jonblog

However, that returns an error:

Conflicting options: --restart and --rm

There are several Docker tickets that say this is sensible, but I don't understand it. I think my meaning is clear: if a container is not running (e.g. on start-up) then I want to start it. If it dies then I expect the container to be removed, and a new fresh one created from the base image. Containers should be immutable anyway - any state I wish to preserve, like media files and logs, will be written to volumes.

So, I thought I should drop the --restart flag, and then use a process manager to stop and start Docker containers. Could I use Monit here? I was hoping to be able to do something like:

CHECK PROCESS jonblog MATCHING jonblog
  START PROGRAM = "/root/docker/jonblog/host-start.sh"
  STOP PROGRAM = "docker stop jon-blog"

However, that checks the system process table, and not docker ps, and so it won't find something matching the specified string. Can I get it to exec docker ps periodically, and match lines in the output?

I'd be happy to use another tool if it proves robust. For example, I find Supervisor a bit heavyweight, but if that is more able to work with Docker, I am willing to use it.

Clarification on --rm

The reason why I want --rm is that during the Dockerisation process, I stop the currently running container, load a new version of the image, and re-run the above script. This means that Docker is being notified of a restart policy for each container. I found that after rebooting the box, I would have 15 or so slightly-differing versions of the app running simultaneously, which is not the intention.

I suppose I could use docker update --restart never on old containers to prevent this happening, but then when my container stops, I am left with it lying around, and I'd just as soon have it auto-delete. I could periodically clean old ones up using some sort of cron job, but that feels a bit hacky given that Docker can do it for me.

Seeking a range of answers

A very helpful comment has suggested I should look into Minikube, which apparently simplifies the setup of Kubernetes, even to the degree a suitably skilled individual can be up and running in five minutes.

I would still quite like to see more lightweight solutions proposed, so I have a range of answers to choose from. As indicated, I would like to find out the answer to whether a process supervisor like Monit would work.

Off the top off my head, I could write a shell loop to write docker ps to a file every five seconds for a minute, and then run that on a Cron every minute. I could then scan that file using grep and the Monit CHECK PROGRAM system check. That's a bit hacky, but is something that I can understand easily if there is a problem with it. Any advances on that suggestion?

I am trying to set up Apache as a frontend proxy to a Docker app. I have done this several times in the past, but for some reason, the new Ubuntu VPS I have is not co-operating. I'm using Apache/2.4.18.

I am using this vhost:

<VirtualHost *:80>
    ServerName blog.example.com

    ProxyPass / http://127.0.0.1:8083/
    ProxyPassReverse / http://127.0.0.1:8083/

</VirtualHost>

I have enabled the necessary modules:

a2enmod proxy
a2enmod proxy_html
a2enmod xml2enc

Apache starts up fine, but I get this error in the logs, plus the standard 500 crash message in the browser:

AH01144: No protocol handler was valid for the URL /. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

This problem is mentioned a lot on the web, but the advice basically reflects the contents of the error message, which is that I need to ensure the proxy modules are loaded. I can confirm 100% they are loaded - in fact, if I replicate the LoadModule commands in my vhost config, Apache won't start, because the modules are already loaded.

They also look loaded from here too:

$ apachectl -M | grep proxy
 proxy_module (shared)
 proxy_html_module (shared)

I can confirm the Docker app is serving an HTTP response (I don't think it's getting that far anyway):

root@agnes:~# wget localhost:8083
--2018-04-02 19:04:46--  http://localhost:8083/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8083... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7 [text/html]
Saving to: ‘index.html’

index.html                                                  100%[=========================================================================================================================================>]       7  --.-KB/s    in 0s      

2018-04-02 19:04:46 (1.20 MB/s) - ‘index.html’ saved [7/7]

root@agnes:~# cat index.html 
Hello

I am rather stuck as to what other problem I could be having. Researching the problem further is not trivial, since search engine results are drowned out by a common answer that does not seem to apply in my case.

A brief description of my set-up: I have 11 Docker containers in a Docker Compose single-server configuration. Some of these containers produce logs, which I write to (mostly individual) on-host volumes. This results in eight log files, which are shared with a Rsyslog container, again via Docker volumes. Finally this container forwards the logs to Papertrail, the cloud-based log aggregator app.

This is all working: I can see Mongo logs, Apache logs, and various app-level logs in near realtime. However, I am conscious that when I restart the logger container, Rsyslog will lose any record of what logs it has pushed, and so will push the whole lot again. Papertrail probably does its own de-duplication, but nevertheless it would be good to remove this duplication.

I am minded to add another on-host volume, so that state is permanent even if the logger container is destroyed and recreated. However, upon checking a staging instance just now, the state directory /var/spool/rsyslog appears to be empty. Thus, I want to ensure Rsyslog really is writing state files before I do this work.

Am I right in thinking that this directory is set by the $WorkDirectory directive?

Here is a lightly-edited snippet of my config file:

# Load Modules
module(load="imfile")

# If we don't use this, we'll get the hostname of the Logger container
$LocalHostName missive-test

# See https://help.papertrailapp.com/kb/configuration/advanced-unix-logging-tips#rsyslog-2
$ActionResumeInterval 10
$ActionQueueSize 100000
$ActionQueueDiscardMark 97500
$ActionQueueHighWaterMark 80000
$ActionQueueType LinkedList
$ActionQueueFileName papertrailqueue
$ActionQueueCheckpointInterval 100
$ActionQueueMaxDiskSpace 2g
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueTimeoutEnqueue 2
$ActionQueueDiscardSeverity 0

# My own addition
$WorkDirectory /var/spool/rsyslog

# See https://help.papertrailapp.com/kb/configuration/encrypting-remote-syslog-with-tls-ssl/#download-root-certificates
$DefaultNetstreamDriverCAFile /etc/papertrail-bundle.pem # trust these CAs
$ActionSendStreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS
$ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
$ActionSendStreamDriverPermittedPeer *.papertrailapp.com

# Slightly edited to obfuscate exact URL
*.* @@logs0.papertrailapp.com:00000

# Use http://www.rsyslog.com/rsyslog-configuration-builder/ to generate new
# log file watchers.

input(type="imfile"
     File="/var/log/missive/controller/socket-listener.log"
     Tag="socket"
     Facility="local0")

The last input directive is an example, there's eight of these, all pretty similar.

Could I be wrong about the location of state data? I've done a few restarts recently, and on each occasion Papertrail did indeed get a 25K new records. So, either it is writing state where I can't see it, or not writing it at all. Am I missing a directive?

I have tried this on Rsyslog 8.26.0 and 8.31.0, and get the same with both versions.

Update 1

What might help is seeing some logs from Rsyslog itself. However, I've checked in /var/log, and there's not a sausage (other than the on-host volumes of the logs I am pushing). Does this need to be turned on explicitly?

After posting this, I've found a command to verify the configuration file, and it looks fine - nothing that looks like it would stop the state files working.

/var # rsyslogd -N1
rsyslogd: version 8.26.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

Update 2

I've now added -dn (debug and foreground) switches to Rsyslog, and am now producing (very verbose) log files. Amongst 300K other lines, I found this:

2227.959643591:main thread    : action 0 queue: starting queue
2227.959648334:main thread    : action 0 queue: is disk-assisted, disk will be used on demand
2227.959656291:main thread    : action 0 queue: params: type 1, enq-only 0, disk assisted 1, spoolDir '/var/spool/rsyslog', maxFileSz 1048576, maxQSize 100000, lqsize 0, pqsize 0, child 0, full delay 40000, light delay 70000, deq batch si
ze 16, high wtrmrk 80000, low wtrmrk 70000, discardmrk 97500, max wrkr 1, min msgs f. wrkr 100000
2227.959661911:main thread    : action 0 queue:Reg: finalizing construction of worker thread pool (numworkerThreads 1)
2227.959666678:main thread    : action 0 queue:Reg/w0: finalizing construction of worker instance data (for 1 actions)
2227.959671472:main thread    : action 0 queue:DAwpool: finalizing construction of worker thread pool (numworkerThreads 1)
2227.959675830:main thread    : action 0 queue:DAwpool/w0: finalizing construction of worker instance data (for 1 actions)
2227.959682905:main thread    : action 0 queue[DA]: starting queue
2227.959687826:main thread    : action 0 queue[DA]: .qi file name is '/var/spool/rsyslog/papertrailqueue.qi', len 37
2227.959691356:main thread    : action 0 queue[DA]: I am a child
2227.959702371:main thread    : action 0 queue[DA]: clean startup, no .qi file found
2227.959706512:main thread    : action 0 queue[DA]: state -2040 reading .qi file - can not read persisted info (if any)
2227.959711817:main thread    : file stream N/A params: flush interval 0, async write 0
2227.959719817:main thread    : file stream N/A params: flush interval 0, async write 0
2227.959724086:main thread    : file stream N/A params: flush interval 0, async write 0
2227.959734668:main thread    : action 0 queue[DA]: params: type 2, enq-only 0, disk assisted 0, spoolDir '/var/spool/rsyslog', maxFileSz 1048576, maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq batch size 8, h
igh wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0
2227.959739181:main thread    : action 0 queue[DA]:Reg: finalizing construction of worker thread pool (numworkerThreads 1)
2227.959743488:main thread    : action 0 queue[DA]:Reg/w0: finalizing construction of worker instance data (for 1 actions)
2227.959747045:main thread    : action 0 queue[DA]: queue finished initialization
2227.959753324:main thread    : action 0 queue: DA queue initialized, disk queue 0x5630e6079ea0
2227.959756804:main thread    : action 0 queue: queue finished initialization
2227.959762404:main thread    : Action builtin:omfwd[0x5630e605cd60]: queue 0x5630e605d1e0 started
2227.959766212:main thread    : Activating Ruleset Queue[0] for Ruleset RSYSLOG_DefaultRuleset
2227.959769728:main thread    : activateMainQueue: mainq cnf obj ptr is 0
2227.959773846:main thread    : main Q: starting queue
2227.959781585:main thread    : main Q: is NOT disk-assisted
2227.959789199:main thread    : main Q: params: type 0, enq-only 0, disk assisted 0, spoolDir '/var/spool/rsyslog', maxFileSz 1048576, maxQSize 100000, lqsize 0, pqsize 0, child 0, full delay 97000, light delay 70000, deq batch size 256, 
high wtrmrk 80000, low wtrmrk 20000, discardmrk 98000, max wrkr 2, min msgs f. wrkr 40000
2227.959793605:main thread    : main Q:Reg: finalizing construction of worker thread pool (numworkerThreads 2)
2227.959797797:main thread    : main Q:Reg/w0: finalizing construction of worker instance data (for 1 actions)
2227.959807534:main thread    : main Q:Reg/w1: finalizing construction of worker instance data (for 1 actions)
2227.959813976:main thread    : main Q: queue finished initialization
2227.959819101:main thread    : Main processing queue is initialized and running
2227.959823416:main thread    : running module imfile with config 0x5630e605bda0, term mode: cooperative/SIGTTIN
2227.959847971:main thread    : configuration 0x5630e6047ae0 activated
2227.959891721:main thread    : rsyslog/glbl: using '127.0.0.1' as localhost IP
2227.959897527:main thread    : signaling new internal message via SIGTTOU
2227.959907993:main thread    : rsyslogd: writing pidfile '/var/run/rsyslogd.pid.tmp'.
2227.960012480:main thread    : rsyslogd: initialization completed, transitioning to regular run mode

This mentions /var/spool/rsyslog a couple of times, so it looks like that configuration is indeed being set correctly. However I am still not really finding anything that would explain why statefiles are not being left after a push to Papertrail.

I will continue to look through the logs to see if anything else jumps out. Comments on this, in the meantime, are welcome.

Update 3

Aha, I have found these lines:

/ # grep -C 1 "NO state file" /var/log/rsyslog.log
2227.960253654:imfile.c       : imfile: trying to open state for '/var/log/missive/controller/socket-listener.log', state file 'imfile-state:-var-log-missive-controller-socket-listener.log'
2227.960264463:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-controller-socket-listener.log) exists for '/var/log/missive/controller/socket-listener.log'
2227.960268423:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/controller/socket-listener.log'
--
2227.971282226:imfile.c       : imfile: trying to open state for '/var/log/missive/transmitter/queue.log', state file 'imfile-state:-var-log-missive-transmitter-queue.log'
2227.971298328:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-transmitter-queue.log) exists for '/var/log/missive/transmitter/queue.log'
2227.971302589:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/transmitter/queue.log'
--
2228.510460963:imfile.c       : imfile: trying to open state for '/var/log/missive/transmitter/worker-manager.log', state file 'imfile-state:-var-log-missive-transmitter-worker-manager.log'
2228.510480060:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-transmitter-worker-manager.log) exists for '/var/log/missive/transmitter/worker-manager.log'
2228.510484619:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/transmitter/worker-manager.log'
--
2228.774990028:imfile.c       : imfile: trying to open state for '/var/log/missive/storage/storage-server.log', state file 'imfile-state:-var-log-missive-storage-storage-server.log'
2228.775007074:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-storage-storage-server.log) exists for '/var/log/missive/storage/storage-server.log'
2228.775011149:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/storage/storage-server.log'
--
2228.797432870:imfile.c       : imfile: trying to open state for '/var/log/missive/outtray/outtray-server.log', state file 'imfile-state:-var-log-missive-outtray-outtray-server.log'
2228.797447810:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-outtray-outtray-server.log) exists for '/var/log/missive/outtray/outtray-server.log'
2228.797452333:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/outtray/outtray-server.log'
--
2228.798755146:imfile.c       : imfile: trying to open state for '/var/log/missive/interface/access.log', state file 'imfile-state:-var-log-missive-interface-access.log'
2228.798767248:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-interface-access.log) exists for '/var/log/missive/interface/access.log'
2228.798771170:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/interface/access.log'
--
2228.802581105:imfile.c       : imfile: trying to open state for '/var/log/missive/interface/error.log', state file 'imfile-state:-var-log-missive-interface-error.log'
2228.802592195:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-interface-error.log) exists for '/var/log/missive/interface/error.log'
2228.802595916:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/interface/error.log'
--
2228.818228008:imfile.c       : imfile: trying to open state for '/var/log/missive/mongo/mongodb.log', state file 'imfile-state:-var-log-missive-mongo-mongodb.log'
2228.818241043:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-mongo-mongodb.log) exists for '/var/log/missive/mongo/mongodb.log'
2228.818244939:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/mongo/mongodb.log'
--
2228.826921335:imfile.c       : imfile: trying to open state for '/var/log/missive/traffic/traefik.log', state file 'imfile-state:-var-log-missive-traffic-traefik.log'
2228.826933834:imfile.c       : imfile: NO state file (/var/spool/rsyslog/imfile-state:-var-log-missive-traffic-traefik.log) exists for '/var/log/missive/traffic/traefik.log'
2228.826937674:imfile.c       : imfile: clean startup withOUT state file for '/var/log/missive/traffic/traefik.log'

These would be expected for a first-time run, but of course it would be expected that the state files would be left behind so that the logs are not pushed again. I will search for these error strings in case that helps, and I'll tail the logs to see if there is anything towards the end that prevents the final state from being recorded.

Update 4

I've tried to add this after my input() definitions, also to no avail:

$InputFilePersistStateInterval 100
$InputRunFileMonitor

This is because the manual says, of the second parameter:

This activates the current monitor. It has no parameters. If you forget this directive, no file monitoring will take place.

However, I think this is referring to a legacy-style log which is declared by $ commands, and not the new-style input() syntax.

Update 5

It occurred to me that my configuration above, which overwrites the default Rsyslog config, might be removing something critical. I have therefore modified the relevant COPY command so that the default rsyslogd.conf persists, and my config becomes additional to that.

The new directives (that are not commented out) are thus:

# rsyslog v5: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark.so # provides --MARK-- message capability
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # kernel logging (formerly provided by rklogd)

# default permissions for all log files.
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# Include configuration files from directory
$IncludeConfig /etc/rsyslog.d/*

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                -/var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  -/var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          -/var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

This is still pushing my logs to Papertrail, and still there are no spool files created.

I was tempted to think this could be a permission issue, but Rsyslog runs as root in Alpine, and I believe queues as implemented as threads in the main process. Thus, they will run as root too, and there should be no problem writing to /var/spool/rsyslog/*.

Update 6

I have asked the readers of the Rsyslog bug tracker whether this might be a bug, perhaps because I'm using Docker.