AWS ElasticBeanstalk docker-thin-pool getting full and causing re-mount of filesystem as read-only?

The .ebextensions suggested by David Ellis worked for me. I'm unable to comment on his answer, but I wanted to add that you can create a new EBS volume instead of using a snapshot. To mount a 40GB EBS volume, I used the following:

option_settings:
  - namespace: aws:autoscaling:launchconfiguration
    option_name: BlockDeviceMappings
    value: /dev/xvdcz=:40:true

See also this documentation, which has an example of mapping a new 100GB EBS volume to /dev/sdh.

The true at the end means "delete on terminate".

I created a new .ebextensions directory containing an ebs.config file with the above code, then zipped that directory together with my Dockerrun.aws.json. Note that the Dockerrun file must be at the top level of the zip, not inside a subdirectory.

To find where Elastic Beanstalk is mounting the volume, use lsblk on the failing instance. It was also /dev/xvdcz for me, so maybe that is the standard.

We got hit by the same issue. The root cause seems to be Docker not mounting its storage engine (thin-provisioned devicemapper by default in Elastic Beanstalk) with the discard options, which in turn fills blocks until it breaks.

I wasn't able to find a definite solution to this, but here is a workaround (see this comment) that I was able to use on affected instances :

docker ps -qa | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ fstrim /proc/Z/root/

I followed the suggestions provided on AWS documentation and everything is working now.
But I had to combine two solutions: increase space and add cronjob to remove old files.
Here's what I did.

First, I changed the volume xvdcz to use 50GB instead of 12GB. That's the storage that we can see on docker system info. In my case it was always full because I upload lots of files every day.

.ebextensions/blockdevice-xvdcz.config

option_settings:
  aws:autoscaling:launchconfiguration:
    BlockDeviceMappings: /dev/xvdcz=:50:true

After I added a cronjob to clean my deleted files that were not used anymore. It was required because Docker still kept them for some reason. In my case once a day it's enough. If you have more uploads than me, you can configure the cronjob to run how many times you need.

.ebextensions/cronjob.config

files:
    "/etc/cron.d/mycron":
        mode: "000644"
        owner: root
        group: root
        content: |
            0 23 * * * root /usr/local/bin/remove_old_files.sh

     "/usr/local/bin/remove_old_files.sh":
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/bin/bash
            docker ps -q | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ sudo fstrim /proc/Z/root/
            exit 0

 commands:
    remove_old_cron:
        command: "rm -f /etc/cron.d/*.bak"

Source: https://docs.aws.amazon.com/pt_br/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-volumes

AWS elasticbeanstalk docker section Environment Configuration documents how it works:

For improved performance, Elastic Beanstalk configures two Amazon EBS storage volumes for your Docker environment's EC2 instances. In addition to the root volume provisioned for all Elastic Beanstalk environments, a second 12GB volume named xvdcz is provisioned for image storage on Docker environments.

If you need more storage space or increased IOPS for Docker images, you can customize the image storage volume by using the BlockDeviceMapping configuration option in the aws:autoscaling:launchconfiguration namespace.

For example, the following configuration file increases the storage volume's size to 100 GB with 500 provisioned IOPS:

Example .ebextensions/blockdevice-xvdcz.config

option_settings:
  aws:autoscaling:launchconfiguration:
    BlockDeviceMappings: /dev/xvdcz=:100::io1:500

If you use the BlockDeviceMappings option to configure additional volumes for your application, you should include a mapping for xvdcz to ensure that it is created. The following example configures two volumes, the image storage volume xvdcz with default settings and an additional 24 GB application volume named sdh:

Example .ebextensions/blockdevice-sdh.config

option_settings:
  aws:autoscaling:launchconfiguration:
    BlockDeviceMappings: /dev/xvdcz=:12:true:gp2,/dev/sdh=:24

I beat my head against this problem for over a day and finally figured it out.

AWS is using the devicemapper backend and creates a 12GB SSD volume that it mounts and uses for the docker images. You have to override the volume it would mount through the elasticbeanstalk extensions concept and deploy via the CLI (there is no way to do this via their GUI, unfortunately).

In the directory you have your Dockerrun.aws.json file, create a directory called .ebextensions and then create a file that ends in .config inside of it. I called mine 01.correctebsvolume.config. Then put the following contents in there:

option_settings: - namespace: aws:autoscaling:launchconfiguration option_name: BlockDeviceMappings value: /dev/xvdcz=snap-066cZZZZZZZZ:40:true:gp2

I ssh'ed into one of my failing boxed directly and found it was mounting /dev/xvdcz. It may be different for you. The snap-066cZZZZZZZZ needs to be a valid snapshot ID. I created an AMI image of the failing instance and used the snapshot that it created in the process. The 40 is how many GB the volume will be, so substitute in what you need. I don't know what the true or gp2 do, but they came from the AMI image block device data, so I kept them.

The magic namespace and option_name come from here in the documentation.

Just increasing the size of the disk will not solve the problem, it will just error later. AWS Recommends mapping a new disk to your container so that any create file /delete file does not affect the Docker Poll Layer.

I am currently looking on it, I've not tested yet but the soluction I come across is having this on m y blockdevice.config

commands:
  01mount:
    command: "mount /dev/sdh /tmp"
option_settings:
  aws:autoscaling:launchconfiguration:
    BlockDeviceMappings: /dev/xvda=:16:true:gp2,/dev/xvdcz=:12:true:gp2,/dev/sdh=:12:true:ephemeral0

Appreciate any comments.