Docker versus Singularity

[hvgazula]

Personal observations/experiences (will be haphazard):

1. What exactly is the end system (os, architecture. and others) we are designing no-brainer tools for? cluster or personal computers? linux/macos/windows or all of them?
2. This question arose because of some personal pain points experienced this week. The models already in the zoo were all built using pip install inside the dockerfile. However, @WilliamAshbee built his environment using conda, which means, we have to find ways to activate the environment as soon as we enter the container. While this has been easier in docker, not so much in singularity. And that too for a sif built from a docker image.
3. Going from docker to singularity is very very time-consuming. Docker is not available on HPC and macOS. Singularity needs VMs to work on windows and macOS. Using Github actions to build docker images is a good idea (thanks to @djarecka) but not so much for debugging because of the lack of caching. Waiting to hear from @dhritimandas about caching on nix/circleCI.

[hvgazula]

By pip install I mean in the shell directly, thus bypassing the need for "activating" environments which otherwise will be difficult as we don't know the "name" the user chose for the environment. One solution is to use conda's "base" environment but again it leads to issue 2. Activating environments inside singularity. FYI, @gaiborjosue and I have been looking into this and there's a lot of info process before we could get to do that.

[dhritimandas]

For 1., I believe it is all of them - the goal being that anyone (beginner or expert, CS person or clinician) wanting to do 3d medical image analysis should be able to use the library nobrainer to meet their goals. The architecture is I believe better defined in the nobrainer roadmap document. While originally meant for Tensorflow/Keras, there has been some discussion about switching between Tensorflow and PyTorch. I am personally keen on trying out Keras 3.0 here which will offer multi-back end support. https://keras.io/keras_core/announcement/

[dhritimandas]

I wrote on Slack about caching using CircleCI/nix/Kaniko. Sharing it here too:

CircleCI: docker layer caching allows you to do this as long as none of the layers in the image change between job runs. dlc will reuse the layer stored in the cache. It can also be used to run jobs in parallel. However, it will not decrease the wall time required to prepare the initial build environment. more info here:
https://circleci.com/docs/docker-layer-caching/
https://circleci.com/blog/config-best-practices-docker-layer-caching/

Nix: I have not personally used this (vs circleci). Nix seems to have a roundabout way of doing this using a mix of nixOS and/or nix-serve and hydra. This is one solution i came across: https://kevincox.ca/2022/01/02/nix-in-docker-caching/

How did it work for you using Github Actions ? Is it that when PRs merge to the default branch, then they do not include the cache ?
For GitHub Actions, I came across Kaniko that allows for layer caching. https://github.com/marketplace/actions/kaniko-action#cache-layers