Push multiple architectures to Docker

[tristan-f-r]

When making the RWR PR, we had the following error in the CI:

2025-06-12T17:11:19.8730341Z PASSED
2025-06-12T17:11:20.3701315Z test/RWR/test_RWR.py::TestRWR::test_rwr 404 Client Error for http+docker://localhost/v1.48/images/docker.io/reedcompbio/rwr:latest/json: Not Found ("No such image: reedcompbio/rwr:latest")
2025-06-12T17:11:20.3703416Z None

However, at the time of writing, there is an image already present:

It seems that the CI is looking for the amd64 architecture, but only the arm architecture is pushed - it would be useful to have more than just the amd64 architecture pushed, and to document that arm-based machines need to also push an amd64 architecture.

[annaritz]

According to this guide, there are a couple ways to do it.

One way is a DIY approach to make separate repos for each architecture, build images locally, and push all images in a manifest. This build is faster than the other approach but requires multiple repos & local builds.

The other way is to use the buildx feature which was introduced in 2019. It is simpler from a deployment perspective but slower and there are more steps to build the image locally.

I'm going to try using buildx for RWR and ST-RWR.

[annaritz]

Having issues with buildx - it seems to hang when asking for authenticating. Here are the commands I tried:

1. Make a docker-container builder (but I don't set it as default).

docker buildx create \
  --name container \
  --driver=docker-container

2. Push rwr for both arm64 and amd64 architectures:

docker buildx build \
 --tag reedcompbio/rwr:v1 \
 --file Dockerfile \
 --platform linux/arm64,linux/amd64 \
 --builder container \
 --push .

The output where it hangs is below (I've let it run for ~3 min - it should definitely get past credential authorization by then). @tristan-f-r can you try these commands and see if you can push multiple architectures? It could be something with reedcompbio (though I can push and pull without any issue).

% docker buildx build \
 --tag reedcompbio/rwr:v1 \
 --file Dockerfile \
 --platform linux/arm64,linux/amd64 \
 --builder container \
 --push .
[+] Building 41.3s (13/14)                                                                                                                 docker-container:container
 => CACHED [linux/arm64 2/4] RUN pip install networkx==2.8.8 numpy==1.26.4 scipy==1.15.2                                                                         0.0s
 => CACHED [linux/arm64 3/4] WORKDIR /RWR                                                                                                                        0.0s
 => CACHED [linux/arm64 4/4] RUN wget https://raw.githubusercontent.com/Reed-CompBio/rwr/9de14ac239df906f5ed0c19b9a42b83f5aa6f296/RWR.py                         0.0s
 => exporting to image                                                                                                                                          40.8s
 => => exporting layers                                                                                                                                          0.0s
 => => exporting manifest sha256:b99093ad58ac3e483c2e7f2fcaa39e5550046bbac45de45b9491529034d5aa1b                                                                0.0s
 => => exporting config sha256:baa2de0cc4687cc869c86243eb4e568b3b834e15e5d086d6cb993fb3216b9652                                                                  0.0s
 => => exporting attestation manifest sha256:a7be2dd54fc69c990391f1a7a9e54d024f0677368a11857ae4ca931eaa478404                                                    0.0s
 => => exporting manifest sha256:6f3e96a23e4a667fec5fdfeedfae6ad748542475bf9c642b8c7f4bedb7fc3a2e                                                                0.0s
 => => exporting config sha256:04aa0328a1c93b4fe3ca58c62da0cf7b34987b619e025bc86f70b9ba8f1cbd36                                                                  0.0s
 => => exporting attestation manifest sha256:2f2883ab9acfabed1a5068b11a61d3372cc36c377fdd12a8da45c2ab88833595                                                    0.0s
 => => exporting manifest list sha256:a209a76d605f83e3b77e2c5e097edb7b1ac26974c91e8517bfb512a4d5bf2663                                                           0.0s
 => => pushing layers                                                                                                                                           40.7s
 => [auth] reedcompbio/rwr:pull,push token for registry-1.docker.io                                                                                              0.0s

[tristan-f-r]

I'll try this 👍

[tristan-f-r]

It worked for me: https://hub.docker.com/r/pubtristanf/rwr/tags [perhaps this could be the same docker-auth issue that @AMINOexe had, where logging out and logging back in fixed it? Not sure what is causing Docker to have these sort of auth auto-invaldiation errors]

[annaritz]

Nope, logging out and back in didn't fix this problem. So weird. We can debug this on Monday. Have a good weekend!

[agitter]

Our release.yml workflow uses the https://github.com/docker/build-push-action action to push the SPRAS Docker image when we release a new version. That action is based on buildx.

Should we work on setting this up through an on-demand GitHub Action? There are workflow_dispatch events that manually run actions. We could set something up to manually build and push an image for multiple architectures as needed on demand (carefully managing image versions).

We support something like this in our Manubot repository to build GitHub pages on demand:

[annaritz]

To wrap up this thread, buildx now works fine from my laptop (still a Docker mystery to us). However, keeping this issue open to retain @agitter's comment about whether we should do an on-demand GitHub Action.