/var/lib/docker/overlay2 -> Disk full

[Martiix]

Description
We recently started using docuum on a few hosts and are seeing that the hosts quickly run out of disk space. We have set the docker usage limit to 500GB as we are hoping to keep some cached images. However, our main problem seems to be with the /var/lib/docker/overlay2 folder which seems to grow out of hand. When I ran docker system prune -a -f on a host, a ton was cleaned(a lot more than 500GB). So the question is, does docuum also clean up /var/lib/docker/overlay2 and is the content considered part of the threashold? The host has a 2TB disk which contains 357GB of other stuff, and the remaining was removed by running docker system prune -a -f.

Instructions to reproduce the bug

Environment information:

• Docuum version: v21.0
• Docker version: 24.0.5, build 24.0.5-0ubuntu1~22.04.1
• OS: Ubuntu:22.04
• Service command: /usr/bin/docuum --threshold "500 GB" --deletion-chunk-size 100

Additional context
Add any other context about the problem here.

[stepchowfun]

Hello, thanks for asking this question. Docuum only cleans up images. My understanding is that /var/lib/docker/overlay2 is used for containers, not images. So my expectation is that this directory is not touched by Docuum or counted against the storage limit.

Perhaps you can check to see if your machines have old containers lying around (e.g., with docker container ls --all)? If so, you can hopefully fix your issue with a cron job to clean them up (or perhaps just start them with docker run --rm in the first place). If the problem doesn't seem to be leftover containers, it could be a bug in Docker.

Unfortunately this does not seem related to Docuum in any way, so I will have to close this issue. But feel free to share an update in this thread in case it's helpful for anyone else! In the future, I think it could be nice for Docuum to clean up containers, volumes, etc. (contributions welcome)—but for now it only cleans up images.

[stepchowfun]

I think I may have spoke too soon. I saw some articles online saying conflicting things about what /var/lib/docker/overlay2 is used for, and now I'm less sure. I'll re-open this issue for now. I should mention that Docuum just runs Docker commands like docker image rm ... and doesn't directly touch the filesystem, so any effect on /var/lib/docker/overlay2 would be indirectly through Docker.

One thing that might be worth looking into is which Docker builder you are using (e.g., buildx?). Docker might be using that directory for the build cache.

[stepchowfun]

I think docker system df should reveal what is using up all that space (containers, volumes, images, or build cache). Can you tun that command on one of the machines that is running out of disk to find out?

[Martiix]

This can then maybe be a feature request instead of a bug. Would be nice to have one good way to clean up everything :) Thanks for investigating. We will go back to our old ways of cleaning up docker for now, but docuum does seem very intersting and promising!

[RMPR]

How much effort would it be to implement that, do you think? Can you point me to relevant parts of the code I can look at and need to potentially modify if needed?

[stepchowfun]

Hi @RMPR, if we just want Docuum to run Docker's built-in pruning commands on a schedule, it would not be that much work (just start a new thread from main). If we want it to do something smarter (like LRU), I imagine it could be a lot more work.