Generic OCI artifact transport?

[cfiderer]

Hello all,
I need to transfer non-image artifacts (that adhere to the OCI Artifact spec) between repositories and the file system (repo -> repo, repo -> file system, file system -> repo), I thought about using the containers/image library.
But unfortunately, the docker:// prefix is the only prefix that I've found for repositories (on the network), and the code related to Docker repositories handles only Docker images (and refuses everything else).
I'd like to ask:

• Are there other transports for OCI artifacts that I've overlooked until now?
• Are there projects on the way that address this feature?

Kind regards, Christian

[vrothberg]

Thanks for reaching out!

Did you try copying an artifact that failed? containers/image is OCI conformant and hence supports artifacts. The transport is called docker:// for historical reasons. We are considering to alias registry:// to it to avoid confusion. That being said, docker:// supports images in the OCI format and Docker v2 schema 1 and schema 2.

[cfiderer]

Hi Valentin,

thank you for your reply. The artifacts that I am talking about are not (OCI or Docker) images, but configuration stuff that adheres to the Artifact Authors Guidance.

When I try to inspect such an artifact, I get an unsupported docker v2s2 media type: error from the manifest.SupportedSchema2MediaType() function. And yes, our layer.mediaType is not one of the known media types for Docker images.

$ skopeo inspect docker://eu.gcr.io/gardener-project/cc/cnudie-test-3/component-descriptors/github.com/gardener/cc-utils@sha256:95df958f73f74e81e14ce83d199e4f3a29dd2957e832c7971a28b302c0b595f4
time="2021-02-17T12:32:35+01:00" level=fatal msg="Error parsing manifest for image: unsupported docker v2s2 media type: \"application/vnd.oci.gardener.cloud.cnudie.component-descriptor.config.v2+yaml+tar\""

A sample manifest is eu.gcr.io/v2/gardener-project/cc/cnudie-test-3/component-descriptors/github.com/gardener/cc-utils/manifests/sha256:95df958f73f74e81e14ce83d199e4f3a29dd2957e832c7971a28b302c0b595f4 (a bit redacted for readability):

$ curl -k -v https://eu.gcr.io/v2/gardener-project/cc/cnudie-test-3/component-descriptors/github.com/gardener/cc-utils/manifests/sha256:95df958f73f74e81e14ce83d199e4f3a29dd2957e832c7971a28b302c0b595f4
> GET /v2/gardener-project/cc/cnudie-test-3/component-descriptors/github.com/gardener/cc-utils/manifests/sha256:95df958f73f74e81e14ce83d199e4f3a29dd2957e832c7971a28b302c0b595f4 HTTP/2
> Host: eu.gcr.io
> user-agent: curl/7.75.0
> accept: */*
>
< HTTP/2 200
< docker-distribution-api-version: registry/2.0
< content-type: application/vnd.docker.distribution.manifest.v2+json
< docker-content-digest: sha256:95df958f73f74e81e14ce83d199e4f3a29dd2957e832c7971a28b302c0b595f4
< content-length: 513
< date: Wed, 17 Feb 2021 10:52:44 GMT
< server: Docker Registry
<
{
    "config": {
        "digest": "sha256:e9615b60fb0495d914d3e478e8714557fc372852baebf894b020e47d72a85471",
        "size": 227,
        "mediaType": "application/vnd.oci.gardener.cloud.cnudie.component-descriptor-metadata.config.v2+json"
    },
    "layers": [
        {
            "digest": "sha256:6c08b425efa1be28a3c6223bb937eeeb1dc5886ac2190e994f705f31ef1ce801",
            "size": 1536,
            "mediaType": "application/vnd.oci.gardener.cloud.cnudie.component-descriptor.config.v2+yaml+tar"
        }
    ],
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "schemaVersion": 2
}

It seems that the non-Docker values in the config.mediaType and layers.[0].mediaType cannot be handled...

[vrothberg]

Thanks for sharing the details, @cfiderer.

I am aware of OCI Artifacts. In a sense they are OCI images but special since they exploit the "extensibility clause" which states that unknown properties must be ignored. That being said, Artifacts must be images with a valid OCI manifest. The media types of configs and layers, however, can differ.

Looking at the upper manifest, I see

"mediaType": "application/vnd.docker.distribution.manifest.v2+json",

It's a Docker v2s2 image but Artifacts are limited to OCI. So the image is neither a valid Docker v2s2 image, nor a valid OCI artifact.

I think containers/image is right to reject it since the config and the layers do not conform with the Docker v2s2 spec:

unsupported docker v2s2 media type: "application/vnd.oci.gardener.cloud.cnudie.component-descriptor.config.v2+yaml+tar""

That doesn't necessarily mean that we cannot change the behavior. Compatibility with other tools is quite often a good reason to relax some checks.

Do Docker and containerd eat the image?

[cfiderer]

I doubt that docker or containerd can process that artifact, because it is not an image. (My local Docker refuses to pull the image with Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:v2'.)

Several registries (e.g. GCR, JFrog Artifactory) however, accept the artifact (store & retrieve).

Our intent is to have an agnostic copy tool that processes the manifest and passes it and the layers unmodified.
So - do we need to change the manifest.mediaType to some other value in order to get such artifacts handled? Or can the checking be relaxed?

[vrothberg]

Our intent is to have an agnostic copy tool that processes the manifest and passes it and the layers unmodified.
So - do we need to change the manifest.mediaType to some other value in order to get such artifacts handled?

Yes. Currently the image claims to be a Docker v2s2 image but it should be an ordinary OCI image just with the specific config and layer media types.

Or can the checking be relaxed?

I have mixed feelings since the image doesn't comply with Docker v2s2. However, the registry seems to happily serve the image which suggests that other tools are more relaxed.

An example OCI manifest looks as follows:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:e2a8572846157b37916ac1c9141b618998dc08b1808c2af4a4931adc479d5922",
    "size": 584
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4c0d98bf9879488e0407f897d9dd4bf758555a78e39675e72b5124ccf12c2580",
      "size": 2811321
    }
  ]
}

An OCI Artifact is conceptually the same but with a different config media type and (optionally) custom layers.

[cfiderer]

Hmmm...

Debugging through the code, I get the impression that the value from the content-type: HTTP header determines how the manifest is handled. If the header is content-type: application/vnd.docker.distribution.manifest.v2+json, I'm already doomed.

Assuming I have no chance to control the content-type: HTTP header - how can I get in to the realm of generic OCI artifacts?

[vrothberg]

I am not sure to understand the question correctly. Can you elaborate?

The underlying problem is that the image is build incorrectly. Docker v2s2 images don't support artifacts. Only images in the OCI format support artifacts.

Docker is also rejecting the image but Docker is supporting artifacts (as containers/image does). The only way I see to fix the issue is to replace the image with a new one in the OCI format. I am quite surprised that gcr is accepting the image.

Note that skopeo inspect **--raw** ... will fetch the manifest (it skips validation):

$ skopeo inspect --raw docker://eu.gcr.io/gardener-project/cc/cnudie-test-3/component-descriptors/github.com/gardener/cc-utils@sha256:95df958f73f74e81e14ce83d199e4f3a29dd2957e832c7971a28b
302c0b595f4|jq .
{
  "config": {
    "digest": "sha256:e9615b60fb0495d914d3e478e8714557fc372852baebf894b020e47d72a85471",
    "size": 227,
    "mediaType": "application/vnd.oci.gardener.cloud.cnudie.component-descriptor-metadata.config.v2+json"
  },
  "layers": [
    {
      "digest": "sha256:6c08b425efa1be28a3c6223bb937eeeb1dc5886ac2190e994f705f31ef1ce801",
      "size": 1536,
      "mediaType": "application/vnd.oci.gardener.cloud.cnudie.component-descriptor.config.v2+yaml+tar"
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "schemaVersion": 2
}

[cfiderer]

OK, we created another test image with correct? mediaTypes:

$ skopeo inspect --raw docker://eu.gcr.io/gardener-project/test/custom:manifest | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.gardener.cnudie.foo.bar.v1+json",
    "size": 1472,
    "digest": "sha256:28f6e27057430ed2a40dbdd50d2736a3f0a295924016e294938110eeb8439818"
  },
  "layers": [
    {
      "mediaType": "application/vnd.gardener.cnudie+text",
      "size": 4,
      "digest": "sha256:494414ded24da13c451b13b424928821351c78fce49f93d9e1b55f102790c206"
    }
  ]
}

But when trying to copy this artifact to a local Docker registry, skopeo fails when attempting to compress the layer "on the fly":

$ skopeo --insecure-policy copy docker://eu.gcr.io/gardener-project/test/custom:manifest docker://ld7622.wdf.sap.corp:50000/d041725/gardener/cnudie:test2
Getting image source signatures
Copying blob 494414ded24d done
time="2021-02-19T18:06:11+01:00" level=fatal msg="Error creating an updated image manifest: Error preparing updated manifest, layer \"sha256:aecc5eb85c4c3096ed907a4fac7545bc0fe0a8f2b4e7175458a32d6fe33a6c3e\": unsupported MIME type for compression: application/vnd.gardener.cnudie+text"
exit status 1

How can I prevent this on-the-fly compression for unknown MIME types?

[vrothberg]

Excellent! Yes, that image is almost correct. The media type of the layer must be changed. The +text implies using a "text" compression which is not covered in the Artifact spec (see https://github.com/opencontainers/artifacts/blob/master/artifact-authors.md#defining-layermediatypes).

[vrothberg]

Oh, there's actually another issue:

"mediaType": "application/vnd.oci.image.manifest.v1+json",

You need to remove this line ^^^

[cfiderer]

Oh, there's actually another issue:

"mediaType": "application/vnd.oci.image.manifest.v1+json",

You need to remove this line ^^^

When reading the OCI Artifacts spec, it refers to the OCI Image Manifest spec, which in turn defines the application/vnd.oci.descriptor.v1+json media type.

Observing that the control over the content-type: HTTP response header is relatively weak (registries tend to use content types at their gusto, but they shouldn't change the manifest itself), how else can I specify the mediaType of my manifest?

I am still struggling to fully understand how a non-image artifact can be correctly handled using the OCI specs. As far as I've observed, I am lost once a vnd.docker.* something kicks in - because then I end up in the realms of the Docker image spec which doesn't upport anything else (than images).

But when registries are free to provide my manifest with whatever content-type: HTTP response header they like, how can I protect my artifact from being "dockerized"?

[vrothberg]

But when registries are free to provide my manifest with whatever content-type: HTTP response header they like, how can I protect my artifact from being "dockerized"?

I don't think that registries are free to do that. The manifest still "looked" like a Docker one as it was explicitly declaring it's manifest media type. Since OCI manifests don't do that, I assume that registry may just have "guessed" the type.

An OCI image will always be served and declared as an OCI image.

Regarding OCI artifacts. Below is a docker.io/library/alpine:latest image converted to an OCI image:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:d2248bd8b5884b9653464153ea67d224b9b216a23b97b0fd8e7bf3c7a643ea02",
    "size": 585
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:ba3557a56b150f9b813f9d02274d62914fd8fce120dd374d9ee17b87cf1d277d",
      "size": 2811657
    }
  ]
}

The "schemaVersion": 2 line and the absence of a top-level media type is making it look like an OCI image. Since all media types of the config and layers are covered by the OCI spec, it is a "real" OCI image (non-artifact).

An OCI artifact looks similar to that but declares a different config media type and (optionally) different layer media types. The OCI spec states that "unknown media types" must be ignored which left enough space for artifacts.

[sftim]

https://oras.land/ covers similar ground to the original issue description, as I understand it. Not the same thing but close.

[mtrmac]

A few updates for the record:

• Registries are expected to return the originally-specified MIME type. If a registry accepts an OCI artifact but returns its manifest with a Docker MIME type, it’s not compliant. (We might, hypothetically, need to adjust for by non-compliant registries to increase interoperability.)

• It used to be the case that OCI manifests didn’t have the mediaType field defined, and it was not supposed to be present. That’s no longer the case, the field is now not required but strongly recommended.

• From the c/image perspective, the major issue is that c/image really doesn’t expect non-image content. It would try to interpret the config as an image config (which is conceptually incorrect but possibly irrelevant), but it would also always try to compress/decompress layers, as indicated by the unsupported MIME type for compression: application/vnd.gardener.cnudie+text error reported above.

  There are some situations where that can be worked around (typically by disabling the layer modification code using something like copy.Options.PreserveDigests / skopeo copy --preserve-digests), but actual support for OCI artifacts requires c/image code to be written, at least enough to recognize OCI artifact as an object where inspecting the config is not meaningful, and modifying the blobs is not desirable.