delete fails in gitlab ci

[Arabus]

what did you do

Run skopeo inside a gitlab ci job to delete a container image at branch merge time e.g.:

skopeo delete --creds "${CI_REGISTRY_USER}":"${CI_REGISTRY_PASSWORD}" "docker://${CI_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"

what did you expect

The image to be marked for deletion / deleted

what happened instead

skopeo reports an error relating to permissions for getting the image manifest before deleting it e.g.:

$ skopeo --debug delete --creds "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" "docker://${IMAGE_NAME}:${IMAGE_TAG}"
 time="2020-03-17T11:11:41Z" level=debug msg="Returning credentials from DockerAuthConfig"
 time="2020-03-17T11:11:41Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
 time="2020-03-17T11:11:41Z" level=debug msg=" Using \"default-docker\" configuration"
 time="2020-03-17T11:11:41Z" level=debug msg="  Using file:///var/lib/containers/sigstore"
 time="2020-03-17T11:11:41Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.syseleven.de"
 time="2020-03-17T11:11:41Z" level=debug msg="GET https://registry.example.com/v2/"
 time="2020-03-17T11:11:41Z" level=debug msg="Ping https://registry.example.com/v2/ status 401"
 time="2020-03-17T11:11:41Z" level=debug msg="GET https://gitlab.example.com/jwt/auth?account=gitlab-ci-token&scope=repository%3Aoperations%2Fservices%2Fcontainer-images%2Fkubectl-helm%3A%2A&service=container_registry"
 time="2020-03-17T11:11:41Z" level=debug msg="Increasing token expiration to: 60 seconds"
 time="2020-03-17T11:11:41Z" level=debug msg="GET https://registry.example.com/v2/operations/services/container-images/kubectl-helm/manifests/fix-image-cleanup"
 time="2020-03-17T11:11:41Z" level=fatal msg="Failed to delete registry.example.com/operations/services/container-images/kubectl-helm:fix-image-cleanup: {\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"authentication required\",\"detail\":[{\"Type\":\"repository\",\"Class\":\"\",\"Name\":\"operations/services/container-images/kubectl-helm\",\"Action\":\"pull\"}]}]}\n (401 Unauthorized)"

I am happy to share these additional information

Apparently the gitlab CI_REGISTRY_USER does not have permission to query for a token that allows '*' actions (as per https://github.com/containers/skopeo/blob/7a0a8c25a26829b827ce33e7845c343395b4f997/vendor/github.com/containers/image/v5/docker/docker_image_src.go#L423) but instead requires that one acquires a token with less permissions i.e., pull,delete.

While it is possible to define a token that has the permission to request '*' actions, this is only possible with a per user account token with api access rights. in addition a token with '*' action rights would enjoy unnecessary permissions opening up possible security problems.

There is an example of the reg tool (https://github.com/genuinetools/reg/blob/65b2c0329da5b239a259f309a7ed8506be1508f6/registry/authchallenge.go#L30) which actually parses the registry response upon initial auth to request the required action permissions instead of '*'. This might be a feasible solution / workaround even for other registries.

Thanks for all the effort you put into this tool, regards

[mtrmac]

Thanks for the report.

In most paths, we can’t rely on scopes from the registry response, because there are some other registries that don’t provide that: containers/image#211 . But we probably can be smarter than we are now.

I’ll move this to the c/image library where the fix would happen.

[Iroxxar]

There is a fix for this. Export an Access-Token with API and read/write permissions for your registry . Import it into your CI/CD-variables and authenticate with it and the appropriate user via skopeo --creds. Keep in mind though that all images with the same SHA256-digest of the tag you're referencing will be deleted. So you migth want to overwrite the image you want to delete with something small and then delete it via skopeo delete, so the digest is different.

[mtrmac]

containers/image#1478 gets us a bit closer: we can now ask for the right token — but only if the we succeed getting the original token. So if GitLab rejects the first "*" actions value, we don’t try again.