ArtifactHub.io Integration

[joeybrown-sf]

readable

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[joeybrown-sf]

I have made substantial changes to this RFC after discussing with an ArtifactHub.io maintainer and doing some more research. I am now proposing to decommission registry.buildpacks.io.

[jkutner]

My biggest bit of feedback is that I think we should tighten up the timeline for cutting over to the ArtifactHub. But I'm less certain about the shutdown of the old registry (either 12 or 18 months)

[edmorley]

Thank you for putting this together! I'm very excited about the prospect of us having fewer services/components to maintain :-)

[joeybrown-sf]

Hey folks I have substantially changed the content of this RFC. I resolved a bunch of open comment threads because I wanted to start a fresh dialog without getting distracted by conversations that were no longer necessarily relevant.

I have spiked out an integration and proved the concept. In the RFC, I now have a simple and concrete plan for this proposed integration and migration.

Please check it out and let me know if you have feedback or if I resolved a still-pertinent question of yours.

[joeybrown-sf]

@jericop, @dmikusa Hey folks I was hoping yall could take a look at this when you get a moment. I have updated the RFC significantly. I have outlined a concrete path to migrating from publishing buildpack index metadata directly to the CNB Registry.

I'm particularly interested in your feedback about the proposed 6 month timeframe to migrate buildpack metadata from your existing publishing process to the new process. I plan on providing some tooling and a walkthrough of migrating and I am more than happy to contribute efforts and work with yall on the Paketo side.

Let me know if there is more information I can provide!

[dmikusa]

This RFC proposes 2 new Artifacts Kinds: cnb-buildpack & cnb-builder.

Is a cnb buildpack type sufficent? We really have two types of buildpacks, component (i.e. I do something) and composite (I bundle together some buildpacks). I don't know what this system is like, but is it flexible enought to have one type that supports both of those needs?

My head goes to wanting to see different things for these. For a component buildpack, I want to see things in the buildpack.toml, like the version and maybe the configuration (if we could do that generically). For the composites, it's about the order groups. What does this composite buildpack pull together? I want to see a list of that. I guess that's why I'm asking, cause in the UI I'd want to see different things based on the type of CNB.

[joeybrown-sf]

Short answer: I think.

I admittedly was playing around with component buildpacks and only glanced at composite buildpacks.

ArtifactHub.io has 2 main UI elements.

The first is a readme that cnb-authors will provide. This readme supports markdown, so you can put whatever markdown you want here.

The second UI element is a widget (that can be embedded externally). It might be really nice to have a standard for sharing composite buildpack hyperlinks in this widget.

This is great feedback. I will mess around with my dev env implementation and give you a more concrete answer this week.

[dmikusa]

I'm not sure I follow how apps could break here. I'm definitely concerned with this case as we really want to avoid Paketo users from having builds fail unexpectedly.

[joeybrown-sf]

Let's say I published a blog 2 years ago and referenced my buildpack joeys-buildpacks/dotnet. There could be users out there that use my buildpack (referenced with the Registry URI). If I migrate my dotnet buildpack to Artifacthub.io and put it in the repository called joeys-buildpacks, everything will continue to work.

But if I do not migrate the joeys-buildpacks/dotnet index to ArtifactHub.io, it will get deleted from the CNB Registry. Any apps that reference this buildpack via the CNB Registry URI will fail to build because Registry URI resolution will fail.

I also want to make sure Paketo users are not impacted. If the Paketo team migrates all their buildpacks, paketo users should see no impact.

I am happy to help the Paketo maintainers with this work.

[dmikusa]

Ah, gotcha. So as long as we follow the steps here there will be no breakage for users. I think that works.

Happy to work together on this. Paketo's always willing to be the Guinea pig for things like this 😄

[dmikusa]

I'm not sure I am following what will happen with the existing data that is in the buildpacks registry now. Is that going to be migrated in some way to the new system? Or is that being put onto buildpack authors?

Paketo has a large history of buildpacks. If we want to keep this accessible to through the registry, what would we need to do? This is definitely something we'd want to do.

[joeybrown-sf]

The existing data will be deleted if it is not migrated to ArtifactHub.io. This RFC puts all the migration work on the Buildpack Authors. The CNB org will not be responsible for migrating any community buildpacks.

If we want to keep this accessible to through the registry, what would we need to do

You would need to create artifacthub-pkg toml files for all existing buildpacks. Check out my proof of concept repo I exported the go buildpacks for paketo.

I have an export script I'm working on that will create artifacthub-pkg.yml files for existing packages (that's what I used to export the go buildpacks). Then going forward, you would be responsible for creating new ones as new versions are published.

[dmikusa]

Ok, so Paketo would need to make a new repo under our Github org. We'd publish all the ArtifactHub data there. Then periodically, ArtifactHub will pick up the data from our repo (and we register this repo with ArtifactHub).

The format doesn't look too bad. I'm sure that we could write a CI job that does that, but I do feel like that is probably something we'll do and a bunch of other people will do also. It seems like it would be a good target for something to share. Perhaps that's something we could work together on and publish through https://github.com/buildpacks/github-actions?

[joeybrown-sf]

Yes you can use a new repo or one you already have. Or you could make a repo for each of your buildpacks, but I think that seems overly tedious and I don't know what the benefit would be to an organization.

I didn't realize we had that github-actions repo existed that's great!

I would love to collaborate on this!