jebpages

While (pretty much) everyone who’s using docker is running it on Linux, and while lots of people run docker on their laptops and desktops, most aren’t running it directly on Linux desktops and laptops. Instead, most individual docker users are relying on some sort of purpose-built Linux distribution running as a virtual machine on their Mac or Windows machine.

However, if you are (like me) running Linux on your desktop, you can run docker containers right on your bare metal, with no virtualization overhead in between. Yay, Desktop Linux!

But wait. If you are (like me) running Fedora Linux on your desktop, and if you (also like me) weren’t thinking about docker and its particular storage needs when you installed Fedora on your machine, you could be in for some perplexing issues or at least crap performance, because of the way that docker storage works on Fedora.

I’ve written about the general issue elsewhere:

…the AUFS backend that started out as Docker’s default storage option, but never made its way into the mainlain Linux kernel, posed a problem for Red Hat and our upstream first, no out-of-tree bits ways.

The settled-upon solution was device mapper thin provisioning, which takes a block storage device to create a pool of space that can be used to create other block devices for Docker containers and images. The device mapper backend can be configured to use direct LVM volumes or you can let Docker create a pair of loopback mounted sparse files to serve as the block devices.

from: Friends Don’t Let Friends Run Docker on Loopback in Production

When you install Fedora on your desktop or laptop, the installer divvies your entire disk up into a small boot partition and a big LVM partition, and then divides that LVM space up into a swap volume that varies in size based on how much RAM you have installed, a root volume of 50GB, and a home volume that takes over whatever’s left.

With no room left for the pool of space that the docker device mapper storage driver needs for containers and images, the storage driver will turn instead to crappily-performing loopback mounted files. Boo!

A Fix

You can cut back the size of the home volume in Fedora without too much trouble. I like to use system-storage-manager to work with my disks:

NOTE: I guess I should add that whenever you’re mucking with your disks, you should make sure you have backups, and so on, but I have resized my own laptop partitions in just this way on more than one occasion, and I’ve tested the steps written here with a VM as I wrote this, so, yeah.

$ sudo dnf install -y system-storage-manager

Next, reboot your machine, and when you get back to the login screen, hit CTRL-ALT-F2 to get to a virtual terminal, and then log in as root. We need to do this in order to unmount the home directory before we shrink it. As root, you can use system-storage-manager to shrink down the home volume. Below I’m shrinking the home volume to 20G, because I’m testing these instructions on a VM with a 100GB drive. Substitute a value that makes sense for your rig.

# umount /home
# ssm resize -s 20G /dev/fedora/home
# reboot

If you’ve already installed and run docker, you’ll need to delete /var/lib/docker, where all of your containers and volumes live, so be prepared to rebuild those.

$ sudo systemctl stop docker
$ sudo rm -rf /var/lib/docker
$ sudo systemctl start docker

When docker starts up again, a script that comes bundled with Fedora’s distribution of docker will check to see that there’s space available in your volume group and will set up your storage correctly. If you want to grow your home volume later, it’s easy to do and doesn’t require unmounting anything. You’ll run the same ssm resize command from above, and swap in your desired volume size.

NOTE: If you’re using docker-engine from docker.com, check out these docs for setting up devicemapper driver correctly by hand.

Starting out right

If you haven’t yet installed Fedora, you can configure your system to accommodate this and other LVM storage scenarios moving forward by making your home volume smaller and modifying your “fedora” volume group from its default “Automatic” size policy to the “As large as possible” policy. This way, all your spare disk space will be ready for new volumes (such as the docker thin pool) or for growing your home or root volumes if you decide that you need the space later on. This is probably how Fedora partitioning should be configured by default, anyway, but it isn’t.

Looking ahead

Finally, there’s another option on the horizon for docker storage on Fedora, an option that doesn’t require partition changes or planning: OverlayFS. I wrote about this in the post I linked above, too, but the TLDR is that OverlayFS and SELinux don’t work together yet, although that’s set to change. Stay tuned.