`docker` transport does not respect per-repo authn challenges if /v2/ does not return 401

[ezekg]

As far as I can tell, Skopeo determines whether authentication is required for a registry based solely on the GET /v2/ response via the presence of a WWW-Authenticate challenge in a 401 Unauthorized response. If this initial "ping" endpoint allows unauthenticated access, Skopeo assumes the entire registry does not require authentication. However, this behavior fails in cases where a registry implements per-repository access controls, where the registry itself, and some repositories, may allow unauthenticated access, while others require authentication. This is common, especially for multi-tenant registries.

This behavior deviates from the Docker CLI and Oras, both which correctly handle per-repository authentication and authorization. Specifically, Docker respects WWW-Authenticate challenges for individual repository endpoints, e.g. /v2/<repo>/manifests/<tag>, even if the GET /v2/ "ping" endpoint previously succeeded without an authentication challenge.

Steps to reproduce

1. Set up a registry with the following behavior:

   • GET /v2/: returns 200 OK without requiring authentication.
   • GET /v2/foo/manifests/latest: requires authentication and responds with 401 Unauthorized and a WWW-Authenticate: Basic challenge.
   • GET /v2/bar/manifests/latest: allows unauthenticated access.

2. Attempt to list tags for the repository that requires authentication using Skopeo:

   skopeo --debug list-tags --creds username:password docker://<registry>/foo

3. Observe that Skopeo does not send or resend the request with an Authorization header, even after receiving a 401 Unauthorized response for /v2/foo/tags/list, even though a challenge was received and credentials were explicitly provided via --creds.

4. Compare this behavior to Docker:

   docker login -u username -p password <registry>
   docker --debug pull <registry>/foo:latest

   Docker correctly resends the request with credentials after receiving the 401 Unauthorized response for /v2/foo/manifests/latest.

5. Compare this behavior to Oras:

   oras login -u username -p password <registry>
   oras pull <registry>/foo:latest --debug

   Oras correctly resends the request with credentials after receiving the 401 Unauthorized response for /v2/foo/manifests/latest.

Expected behavior

Skopeo should respect WWW-Authenticate challenges for individual repository endpoints when and if they occur.

Actual behavior

Skopeo assumes that the registry does not require authentication if the initial GET /v2/ "ping" endpoint responds with 200 OK, and does not attempt to resend the failed request even thought an authentication challenge is present in the response.

Debug logs

skopeo --debug list-tags --creds username:password docker://<registry>/foo
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying to access "<registry>/foo:latest"      
DEBU[0000] Returning credentials for <registry>/foo from DockerAuthConfig
DEBU[0000]  No signature storage configuration found for <registry>/foo:latest, using built-in default file:///home/$USER/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/<registry>
DEBU[0000] GET <registry>/v2/
DEBU[0000] Ping <registry>/v2/ status 200 
DEBU[0000] GET <registry>/v2/foo/manifests/latest
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Accessing "<registry>/foo:latest" failed: reading manifest latest in <registry>/foo: unauthorized:     
FATA[0000] Error parsing image name "docker://<registry>/foo": reading manifest latest in <registry>/foo: unauthorized:

Related issues

• docker: mimic docker upstream registry authentication image#211
• Document and possibly clean up gcr.io authentication support image#200
• docker: fix unauthenticated pulls from gcr.io image#195
• docker: set basic auth when requesting bearer token image#191

Relevant comment in containers/image#195

[…]

I guess it makes sense (all of the server may require authentication, but an individual repository does not need it), though it does seem to be in violation of https://github.com/docker/distribution/blob/master/docs/spec/api.md#api-version-check.

Anyway, ultimately it means that this PR is necessary; most of the objections, if any, should be directed at containers/skopeo#191, or perhaps dockerClient should not use the /v2/ ping to get expected auth parameters at all, and only interpret WWW-Authenticate from the actual API endpoints it needs to use (which would be a much bigger refactoring).

Emphasis mine. Fixed link reference is here.

[mtrmac]

I’m afraid just a quick note:

As far as I can tell, Skopeo determines whether authentication is required for a registry based solely on the GET /v2/ response via the presence of a WWW-Authenticate challenge in a 401 Unauthorized response.

Yes.

I agree that it’s, at the very least, conceptually unclean. But, also, the history (e.g. around the linked issues) was painful, and I’m afraid we don’t have a practical way to test proposed changes against all the registries that we might encounter in the wild.

So, I’m rather reluctant to make changes here, though I expect it should be possible to do something ~clearly low-risk.

Anyway, transferring to the c/image repo, where the changes would happen.