Introduce `artifacts:` section in DVC and make it work with GTO

[aguschin]

Summary / Background

To make working with Model Registry (including Studio MR) more accessible for DVC users, we merge artifacts.yaml into DVC. More detail discussion on this iterative/gto#337

Release scope

Described in iterative/mlem.ai#323, but summarizing it here. For now I'm excluding what we considered to be extra features (they can be found in docs proposal marked as [extra for now]):

• adding top-level registry: section and exposing for API user (Studio) Add top-level artifacts: section #9220
• allow it to be read from a file instead of writing it in dvc.yaml completely, e.g. registry: artifacts.yaml Add top-level artifacts: section #9220
• make_dvcyaml: Create artifacts section dvclive#519
• updating example-gto repo and other fixture repos for Studio BE example-gto: remove annotate and move artifacts to dvc.yaml iterative/example-repos-dev#192
• DVC API to add an artifact Add API to add artifacts #9345
• switching Studio to new mechanics https://github.com/iterative/studio/issues/5704
• removing artifacts.yaml part from GTO Remove artifacts.yaml part in favor of artifacts: section in DVC iterative/gto#346
• updating GTO GH action to work with new annotations format https://github.com/iterative/gto-action/tree/v2
• updating docs Model registry docs dvc.org#4423
• update https://github.com/iterative/dvcyaml-schema Add artifacts: section to the schema dvcyaml-schema#36
• communication with customers/users to migrate from old MR to new MR

Follow-ups after release (p1)

• make a video showing e2e workflow (add/log->compare->register->assign/promote->download->deploy) to publish to YouTube/other channels
• Improve CI/CD example to trigger upon correct Git tags only iterative/gto#354
• update example repos: demo-bank-customer-churn, example-get-started-experiments, demo-fashion, etc
• follow-up on this for GTO users in DVC+GTO integration iterative/gto#337
• update GTO action to be able to download artifacts Make it possible to download DVC artifact iterative/gto-action#17

Follow-ups after release (p2)

• implementing dvc get $REPO myartifact#prod Download models (type: model) with dvc get #9100
• implementing way to get #prod or @latest with DVC API
• consider Using dvc exp workflow together with MR #9102
• support dvc import for artifacts same way we should support dvc get already

Related issues

• the artifacts piece @skshetry implemented already Meta information for data #8214 add metadata fields: label, type to data #8232

[shcheklein]

Thanks @aguschin, sounds like a good plan, overall. A few questions:

allow it to be read from a file instead of writing it in dvc.yaml completely, e.g. registry: artifacts.yaml

can we cut the scope? is it needed?

updating docs

does it make sense to start with this?

Do we need to include an item that simplifies GTO? (removes the registry files mechanics).

[aguschin]

can we cut the scope? is it needed?

Maybe we can, but I'm trying to find a way to not break current users workflows. GTO don't have that many users, and I wouldn't like to lose them. This item can be postponed though - there will be some time before we implement this, and before we release it. So we can prioritize other things and get back to it before the release. Not mentioning some users asking for that specifically.

Re how we can avoid breaking user workflows though. When we support registry: artifacts.yaml right away, this won't add it to dvc.yaml still, so in the existing commits this part won't be discovered automatically. But you can add this to dvc.yaml and delay migration, while still using GTO for some time (since artifacts.yaml is still there). This will allow a smooth transition.

does it make sense to start with this?

IIRC, @dberenbaum wanted to take it and add to DVC docs? Although I had an impression there are some WIP changes proposed by @daavoo. Happy to help there if needed - writing docs is so much faster for me than implementing this in DVC (for now 😅 ).

[dberenbaum]

updating docs

does it make sense to start with this?

@shcheklein I added treeverse/dvc.org#4423 to discuss docs. Since @aguschin already drafted iterative/mlem.ai#323, is there something missing from there that you'd like to see before implementation?

allow it to be read from a file instead of writing it in dvc.yaml completely, e.g. registry: artifacts.yaml

can we cut the scope? is it needed?

I asked @aguschin the same thing 😄, and he explained that he will try to do it to make it easier for existing users to transition if it's low effort. I do worry about documenting this and making dvc.yaml even more complex, but I think we could try to minimize this. I would even consider not documenting it for now if it's strictly a legacy thing to help existing GTO users.

[shcheklein]

can we cut the scope? is it needed?

Folks, we need to be extra extra mindful about complexity and scope. Can we help the existing users (and we don't have many, right?) to transition to the new format manually, of with some small written instruction?

Complicating parsing in Studion, dvc.yaml format, etc - doesn't unfortunately sound like a very easy thing to do.

I feel your concerns and I like the deep care to the users though @aguschin !!

is there something missing from there that you'd like to see before implementation?

I don't know (your call), I was asking specifically because it sometimes uncovers more things. If you feel that it's stable already, we get sense for all the changes, etc - we good to go then.

[dberenbaum]

@aguschin We are aiming to include this as part of a 3.0 release at the end of this quarter. Do you expect that we can get all of the items above done? Do you think there is more that needs to be added before release?