How do I use OverlayFS?

Edit: Since writing this answer, some things have changed in overlayfs, namely the addition of a required parameter workdir, see totti's answer below for a detailed description of this new parameter.

I finally managed to find it. I found references to it in the kernel source, but for some reason it doesn't appear in the git tree on kernel.org. But! If you pull the Ubuntu kernel source like this: apt-get source linux-image-3.0.0-16-generic you can find it in linux-3.0.0/Documentation/overlayfs.txt. It is also available in the linux-doc package in /usr/share/doc/linux-doc/filesystems/overlayfs.txt.gz.

As the actual help documentation is more of a "how it works" instead of a "how to mount with it," here's a brief rundown (there is one example in the kernel documentation):

mount -t overlayfs -o [mount options] overlayfs [mountpoint for merged system]

Where [mount options] can be:

• lowerdir=somedir: lowerdir is the directory you're going to lay your new filesystem over, if there are duplicates these get overwritten by (actually, hidden in favor of) upperdir's version
• upperdir=somedir: upperdir is the directory you want to overlay lowerdir with. If duplicate filenames exist in lowerdir and upperdir, upperdir's version takes precedence.
• standard mount options. The only one I've seen from code is ro/rw, but you can experiment.

One thing that confused me at first, so I should probably clarify, is that mounting an overlayfs does not actually mount a filesystem. I was trying to mount a squashfs filesystem using an overlayfs mount, but that's not how it works. You must first mount the (in my case squashfs) filesystem to an arbitrary directory, then use overlayfs to merge the mount point (a directory) and another directory onto a tertiary directory (the overlayfs mount point)(edit: this "tertiary" directory can actually be the upperdir= directory). The tertiary directory is where you will see the merged filesystems (or directory trees - it's flexible).

Example 1, overlaying the root filesystem

I've been working on an Ubuntu hybrid boot disk where the base Ubuntu system exists as filesystem.squashfs and I have files called ubuntu.overlay kubuntu.overlay xubuntu.overlay and lubuntu.overlay. The .overlay files are base installs of said systems with the contents of filesystem.squashfs pruned (to save space). Then I modified the init scripts to overlay the correct distro's .overlay file (from a boot parameter) using overlayfs and the above options and it works like a charm!

These are the lines that I used in my init scripts (once all variables are translated):

mkdir -p /overlay
mount -t squashfs /cdrom/casper/ubuntu.overlay /overlay
mount -t overlayfs -o lowerdir=/filesystem.squashfs,upperdir=/overlay overlayfs /

Note that filesystem.squashfs above is a directory created by casper, not a file.

These three statements create an /overlay directory, mount a squashfs filesystem on the /overlay directory and then use OverlayFS to essentially merge the contents of /overlay over /.

Example 2, transparent merging of two directories

In the process of re-building my live USB for each release, I use OverlayFS to save a bunch of time. I start off with a directory called ubuntu-base which contains the contents of the ubuntu-core image which is the most basic install. I will then create directories called ubuntu, kubuntu, lubuntu, and xubuntu.

Then, I use OverlayFS to make the files from the ubuntu-base show up in the individual directories. I would use something like this:

mount -t overlayfs -o lowerdir=ubuntu-base,upperdir=kubuntu overlayfs kubuntu

This makes the files from ubuntu-base show up in the kubuntu folder. Then, I can chroot to the kubuntu folder and do something like apt-get install kubuntu-desktop. Any changes that are made while in this OverlayFS mount will remain in the upper directory, in this case the kubuntu folder. Then, once I unmount the OverlayFS mount the files that really exist in ubuntu-base but are "mirrored" into the kubuntu folder vanish unless they have been changed. This keeps me from having to have multiple copies of the files in ubuntu-base while still being able to use them as if they physically exist in each location.

From https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt:

Upper and Lower

An overlay filesystem combines two filesystems - an 'upper' filesystem and a 'lower' filesystem. When a name exists in both filesystems, the object in the 'upper' filesystem is visible while the object in the 'lower' filesystem is either hidden or, in the case of directories, merged with the 'upper' object.

It would be more correct to refer to an upper and lower 'directory tree' rather than 'filesystem' as it is quite possible for both directory trees to be in the same filesystem and there is no requirement that the root of a filesystem be given for either upper or lower.

The lower filesystem can be any filesystem supported by Linux and does not need to be writable. The lower filesystem can even be another overlayfs. The upper filesystem will normally be writable and if it is it must support the creation of trusted.* extended attributes, and must provide valid d_type in readdir responses, so NFS is not suitable.

A read-only overlay of two read-only filesystems may use any filesystem type.

Directories

Overlaying mainly involves directories. If a given name appears in both upper and lower filesystems and refers to a non-directory in either, then the lower object is hidden - the name refers only to the upper object.

Where both upper and lower objects are directories, a merged directory is formed.

At mount time, the two directories given as mount options "lowerdir" and "upperdir" are combined into a merged directory:

mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,workdir=/work /merged

The "workdir" needs to be an empty directory on the same filesystem as upperdir.

Then whenever a lookup is requested in such a merged directory, the lookup is performed in each actual directory and the combined result is cached in the dentry belonging to the overlay filesystem. If both actual lookups find directories, both are stored and a merged directory is created, otherwise only one is stored: the upper if it exists, else the lower.

Only the lists of names from directories are merged. Other content such as metadata and extended attributes are reported for the upper directory only. These attributes of the lower directory are hidden.

I have extended these artikels to include a Script for overlayfs that sets up a read-only root fs.

• English: https://help.ubuntu.com/community/aufsRootFileSystemOnUsbFlash
• German: http://wiki.ubuntuusers.de/Nur-Lesen_Root-Dateisystem

Hope it helps.

Minimal runnable example

# Create the filesystems.
dd if=/dev/zero of=lower.ext4 bs=1024 count=102400
mkfs -t ext4 lower.ext4
cp lower.ext4 upper.ext4
mkdir lower upper overlay
sudo mount lower.ext4 lower
sudo mount upper.ext4 upper
sudo chown "$USER:$USER" lower upper
printf lower-content > lower/lower-file
# Upper and work must be on the same filesystem.
mkdir upper/upper upper/work
printf upper-content > upper/upper/upper-file
# Work must be empty. E.g. this would be bad:
#printf work-content > upper/work/work-file
# Make the lower readonly to show that that is possible:
# writes actually end up on the upper filesystem.
sudo mount -o remount,ro lower.ext4 lower

# Create the overlay mount.
sudo mount \
  -t overlay \
  -o lowerdir=lower,upperdir=upper/upper,workdir=upper/work \
  none \
  overlay \
;

# Interact with the mount.
printf 'overlay-content' > overlay/overlay-file
ls lower upper/upper upper/work overlay

# Write to underlying directories while mounted
# gives undefined behaviour.
#printf lower-content-2 > lower/lower-file-2
#printf upper-content-2 > upper/upper-file-2

# Unmount the overlay and observe state.
sudo umount overlay
ls lower upper/upper upper/work

# Cleanup.
sudo umount upper lower

GitHub upstream.

Output of the first ls with the mount:

lower:
lost+found  lower-file

overlay:
lost+found  lower-file  overlay-file  upper-file

upper/upper:
overlay-file  upper-file

upper/work:
work

Output of the second ls without the mount:

lower:
lost+found  lower-file

upper/upper:
overlay-file  upper-file

upper/work:
work

Interpretation:

• lower: was unchanged after writing to overlay
• upper: received the modification to overlay
• overlay: shows files from both upper and lower
• work: contains some random content (a work/ directory) we should not care about

Example adapted from: Example OverlayFS Usage

Here is a more complex example with multiple lower layers: Overlayfs reload with multiple layers (migration away from aufs)

Tested on Ubuntu 18.04, Linux kernel 4.15.0.

Mounting OverlayFS in fstab

Above answers have explained how to mount overlay filesystem from cmd line and/or script. It is also possible to mount OverlayFS from fstab.

There is inherent problem with mounting OverlayFS. Since we mount directories, those directories must be present and/or ready. Present means that filesystem on which particular directory is located is already mounted, while ready means that particular directory is "populated". Example of later is mounting of ISO image to particular directory, or mounting network share (of any type).

Mounting directory which is not present, will fail, while mounting not yet "populated" directory will not fail, since OverlayFS has no way to know, if directory is ready.

When mounting filesystems from fstab, that is particularly a problem, since all mounting is done in parallel, thus we can not ensure that everything is prepared for OverlayFS.

However, if Linux distribution uses systemd, than mounting of filesystems is handled by it. Systemd reads fstab file, and creates mount units on the fly. After that all mounting is processed by systemd. Systemd unit files have options: require, after, before, which can be used to control the order and dependency of mounts defined in fstab, thus ensuring that mounting of OverlayFS will not fail (at least not, because of wrong order).

To set order/dependency of mounts in fstab file, we will declare systemd option "require" using syntax: x-systemd.require. Argument for this option is mount point of the mount which should be successfully mounted before given mount (defined in fstab).

Example:

To show an example (and syntax of fstab entries), we will show case, where we will overlay directory on which iso image is mounted, while this particular image file will be stored on separate hdd, which will have to be mounted before we access image file. Thus forming required ordered of events:

mount hdd -> mount iso image -> mount OverlayFS

For given case fstab entries would look like:

# 1. mount hdd partition
# 2. mount iso image from hdd partition, but only if/after hdd is mounted
# 3. overlay iso image, but only if/after iso image is mounted
#
/dev/hdb1            /mnt/hdd      ext4     errors=remount-ro                 0  2  
/mnt/hdd/linux.iso   /mnt/iso      auto     x-systemd.requires=/mnt/hdd,ro    0  0
overlay              /mnt/merged   overlay  x-systemd.requires=/mnt/iso,lowerdir=/mnt/iso,upperdir=/overlay/lower,workdir=/overlay/working  0 0
                                                                            

If any mounts in chain fail, dependent one(s) will not be executed.

Note:

On Arch Linux Wiki there is example of using fstab to mount OverlayFS in which following options are used:

noauto,x-systemd.automount

This will have effect, that overlay filesystem will be simply not mounted at boot time (noauto option), avoiding possible failure if previous mounts are not ready. On first access to overlay directory, the filesystem will be mounted (x-systemd.automount option). This approach has two weaknesses:

1. If needed lower/upper directories are not ready this mount will fail.
2. Mount will succeed even if lower/upper directories are not populated (ISO is not mounted f.e.)

We can avoid later, by adding x-systemd.require to the options list.