Summary

The web content discusses debugging Docker Overlay2 storage issues, focusing on identifying and resolving excessive disk space usage by Docker containers.

Abstract

The article addresses a common problem encountered by Docker users: the Overlay2 folder consuming significant disk space, leading to system storage issues. It explains the Overlay2 file system as a Union File System used by Docker, which can result in multiple file copies and large storage usage. The author provides diagnostic commands to identify which containers are using the most space, such as du -sh /var/lib/docker/overlay2 | sort -h and docker inspect. Misconfigured containers that write logs or data files directly to the container's file system, rather than using volumes, are identified as a primary cause of bloat. The article suggests using volumes to prevent size increase and ensure data persistence outside the container lifecycle. It also recommends multi-stage Dockerfiles to minimize the size of production images and highlights the importance of cleaning up unused images and volumes with commands like docker system prune -a --volumes. Finally, it advises monitoring docker volumes for rapid growth due to application data like logs or database shards, which may require configuration changes to resolve.

Opinions

The author implies that understanding the Overlay2 file system is crucial for debugging disk space issues effectively.
It is suggested that using du and sort commands is a practical approach to diagnose disk space usage by Docker containers.
The article emphasizes the importance of proper container configuration to avoid excessive storage consumption, advocating for the use of volumes for log files and data.
There is an opinion that multi-stage Dockerfiles contribute to leaner container images by separating build and production stages, thus reducing the potential for bloated containers.
The author conveys that regular maintenance, such as pruning unused Docker images and volumes, is essential for managing disk space.
It is mentioned that even with proper configuration, applications may still cause volumes to grow rapidly, indicating a need for careful application design and monitoring.

Debugging Docker Overlay2 Out of Space

“My docker container is using 80+ GB of disk space! My system is out of disk space! Help!” -a cry for help by someone experiencing a run away container eating up disk space. A common issue with using docker is the /var/lib/docker/overlay2 folder may take up gigabytes of space which will eventually cause an out of disk space issue on the host system. A quick google search reveals pieces of information but trying to debug this issue is difficult because it require understanding how the overlay2 file system works and tracing which container the overlay folder belongs to.

What is overlay2?

A detailed explanation of what the docker overlay2 can be found on docker’s website. The short explanation is it’s one of the choices for the container file system that uses a Union File System. In laymen terms, the /var/lib/docker/overlay2 folder represents a file system that every file that the container read or writes can be found within. Since it is a Union File System, the container does not make direct modifications to any original files so there may be two copies of the same file. The base image (original files) are stored in a “lowerdir”, the modified files are stored in “upperdir”, and finally combination of the two is the “merged” folder. For example, there is a file called foo that is 50mb in the “lowerdir”. The container writes to the file which causes a modified file containing the changes to exist in the “upperdir”.

Use the OverlayFS storage driver

OverlayFS is a modern union filesystem that is similar to AUFS, but faster and with a simpler implementation. Docker…

docs.docker.com

How to investigate

The commands below are extremely useful to diagnose and narrow down which containers are using a lot of disk space.

du -sh /var/lib/docker/overlay2 | sort 10

Du and sort will show the top 10 folders inside the overlay2 that are using the most space.

docker inspect docker inspect — format=’{{.GraphDriver.Data}}’

Docker inspect shows which overlays folder the docker image is using. These can be found under the GraphicDriver’s Data section.

docker container diff

The docker container diff shows what files are being written and accessed by a specific container.

The Usual Culprits and How to Fix

A misconfigured Docker container

One of the most common issue is misconfigured docker containers where the container is writing files into the container’s filesystem such as log files or a database’s data files. The correct way for container log files and data files is being written into a docker volume.

In addition, volumes are often a better choice than persisting data in a container’s writable layer, because a volume does not increase the size of the containers using it, and the volume’s contents exist outside the lifecycle of a given container.

Use volumes

Volumes are the preferred mechanism for persisting data generated by and used by Docker containers. While bind mounts…

docs.docker.com

A bloated container with lots of layers or a large base image

The next common issue comes from a badly written dockerfile that causes the container to be larger than it needs to be. When loading a docker container, each image layer is loaded into the overlay folder. For example, having a single stage may cause issues such as the container having leftover build artifacts or supporting compilation libraries in the container image. Using a multi-stage dockerfile helps with this problem since there is a separation between the build stage and production stage. The final folder or binary is copied over to the production stage while leaving behind all the unnecessary files, which keeps the container lean.

Leftover containers on the system

Sometimes there are leftover docker images of previously ran docker containers that have not been cleaned up. Running the command below will show how much disk space the containers, volumes, and build cache are using.

docker system df

The unused images can be removed by running the command below, which will remove all data of non-running containers.

docker system prune -a --volumes

A docker volume that is growing rapidly

Lastly, even when everything was configured correctly, there could be a docker container that is writing large amounts of data such as logs or database shards into a docker volume. This is an application specific issue that would need to be resolved by changing the application configuration. Running this command below will show verbose details about the disk space usage of docker.

docker system df -v

After identifying the culprit, if the issue is archival related, it’ll require modifying the application configuration to turn off the logs or put them into an archival server/storage.