Questions[amazon-efs](server)

I noticed a strange behaviour on the setup that I created on AWS (eu-south-1 region) to evaluate the migration from our current provider. I want to deploy 1-n EC2 instances that serves web requests through Apache, serving files stored on a common EFS volume. I already implemented and tested the RDS part, the PHP performance, the ElasticCache integration and so on. However, I noticed a 5 seconds delay on sporadic requests, a delay that seemed very deterministic and too much near the 5 seconds value. The EFS volume is in burst-mode, the credits are high (2T) and the percentage of usage is really low, so this should not be the problem.

I mounted the EFS volume with the suggested options, both with the "EFS mount helper" and the "NFS Client", nothing changed. So, I tried to restart from the scratch, installing just the default Apache web server (even tried with Nginx with similar results) and mounting the EFS volume and benchmarking from another EC2 instance with the following command:

siege -c 2 -r 20 -b http://35.152.48.17/efs-mount-point/efs-test/logo.png

With Ubuntu 18.04 and Ubuntu 20.04, the longest transaction is always above 5 seconds (5.12 - 5.42 seconds). With AmiLinux instead, the longest transaction is fast enough (0.15 seconds). Interestingly, if I lower the parallel clients from 2 to 1:

siege -c 1 -r 20 -b http://35.152.48.17/efs-mount-point/efs-test/logo.png

the longest transaction is ok also on Ubuntu, even if I let "siege" run for more repetitions:

siege -c 1 -r 10000 -b http://35.152.48.17/efs-mount-point/efs-test/logo.png

However, if we remove the EFS variable on the Ubuntu and we serve the files from the local EBS, the longest transaction is blazing fast (few milliseconds), so the problem arises only on EFS with Ubuntu (both 18.04 and 20.04). Maybe the suggested mounting options works for AmiLinux but lack something for Ubuntu AMIs?

The repro steps are so easy that it feels so strange to me:

1. choose the Ubuntu 18.04 AMI;
2. mount the EFS volume (either with the "EFS mount helper" or with the NFS client);
3. install Apache updating just the serving directory.

Any suggestion?

Question: Why does an NFS mount using the sync option cause nginx and/or Chrome (and only Chrome; not Safari, Firefox, Edge or curl) to chunk or restart the file upload?

Some background:

I have nginx in front of a server cluster. We added file uploading and we're using nginx to handle the upload and then pass the uploaded file body via its X-FILE header.

Until recently we were only processing uploads on the nginx server, due to the issues with other application servers in the cluster being unable to access the client_body_temp_path (/tmp in this case).

We run on AWS, so I provisioned an EFS (aka NFS) share and mounted this at /mnt/efs and then changed nginx client_body_temp_path to point at /mnt/efs/tmp.

I mounted the NFS share on all the application servers and they can now access the file uploads on /mnt/efs/tmp, but the uploaded files are often zero-length when they're initially read by the other clients, due to the NFS asynchronous writes.

I changed the NFS mount just on the nginx server so it's mounted with sync:

Initial mount:

example.eu-west-1.amazonaws.com:/ /mnt/efs nfs4 vers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev,nofail

New mount:

example.eu-west-1.amazonaws.com:/ /mnt/efs nfs4 vers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev,nofail,sync

I remounted the share and tried some uploads and happy days, the other NFS clients see the non-empty files in /mnt/efs/tmp.

But with some more testing it turns out that uploaded files over about 10M get restarted or chunked by Chrome.

What happens:

nginx starts to receive the upload (in the video below, I'm sending a 32.7MB file), but at 6.4MB, a new upload starts and then eventually nginx decides that the client's stopped sending, so issues a 499 which causes Chrome to abort the upload and it all fails. You can see all this happening here, as I watch the files coming in to /mnt/efs/tmp:

http://jay.gooby.org/media/video/chrome%20failing.mp4

The upload initially comes in as 0000000006 but then when it gets to 6.4M, a second upload 0000000007 starts (but 0000000006 continues to 12M), and then when 0000000007 hits 6.4, a third, 0000000008 starts (0000000007 continues to 12M). Once 0000000008 hits 12M ngingx issues the 499 and Chrome aborts the upload.

Other useragents don't have this problem. I'm able to upload the same file with Safari, Firefox and curl. Chrome on Mac and Windows both have the same issue.

Here's curl sending the file with no problems: it arrives as 0000000010 single 32M file:

http://jay.gooby.org/media/video/curl%20working.mp4

After tearing my hair out for a bit, I removed the sync option from the NFS mount on the nginx server and lo and behold, files uploaded with Chrome arrive in one piece as a single file. So what gives?

nginx is behind an Elastic IP, but not an ELB, and the upload is over http2.

nginx is compiled with --with-threads option

Update 1: That 6.4mb value seemed significant, as that's the point that each new upload chunk began/restarted. 6.4mb is 51200000 bits which seems like a quite specific value...

The initial bytes of the three files are identical, so it's as if Chrome decided to start resending at the 6.4mb point:

I am using EKS with Kubernetes version 1.15 and when I create a Storageclass, Persistent-Volume, Persistent-Volume-Claim, and Deployment the pod fails with:

Warning  FailedAttachVolume  71s (x2 over 3m11s)  attachdetach-controller              AttachVolume.Attach failed for volume "efs-pv" : attachment timeout for volume fs-<volume>
Warning  FailedMount         53s (x2 over 3m8s)   kubelet, ip-<ip-address>.ec2.internal  Unable to mount volumes for pod "influxdb-deployment-555f4c8b94-mldfs_default(2525d10b-e30b-4c4c-893e-10971e0c683e)": timeout expired waiting for volumes to attach or mount for pod "default"/"influxdb-deployment-555f4c8b94-mldfs". list of unmounted volumes=[persistent-storage]. list of unattached volumes=[persistent-storage]

However when I try the same without building the Persistent-Volume it is successful, and creates its own that seemingly skips CSI. This is what I am working with:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: influxdb-deployment
spec:
  selector:
    matchLabels:
      app: influxdb
  replicas: 1
  template:
    metadata:
      labels:
        app: influxdb
    spec:
      containers:
      - name: influxdb
        image: influxdb:1.7.10-alpine
        ports:
        - containerPort: 8086
        volumeMounts:
        - name: persistent-storage
          mountPath: /var/lib/influx
      volumes:
      - name: persistent-storage
        persistentVolumeClaim:
          claimName: efs-claim

storageclass.yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
reclaimPolicy: Retain

persistent-volume.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-<volume-id>

persistent-volume-claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

Any idea on what is happening?

Based on my understanding of AWS documentation it appears that the only way to encrypt at rest existing EFS instances with some data is to create new EFS instances with encryption enabled and copy the files from unencrypted EFS to encrypted EFS and alter mount points if any.

Can anybody confirm that is the case?

I have a server that is outside of AWS. I'd like to be able to mount an EFS volume to it, but I am not sure if that is possible.

Perhaps if you create a VPC, and you create a tunnel over VPN?

Does anybody know if this is possible?