From 240765289b29780b1bb7c3881e487848a2c478d9 Mon Sep 17 00:00:00 2001 From: Chris Date: Mon, 4 Sep 2023 11:29:29 +0100 Subject: [PATCH 1/3] Add initial docs about custom stacks --- README.md | 2 +- docs/index.md | 2 +- docs/resource-stacks.md | 45 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4497d4b8..aac09476 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ We've put a lot of thought into what our users — data scientists, ML engineers * Somewhere to deploy and serve models in a way that scales with your application needs. * The ability to monitor models for things like drift and bias. -Matcha is still in alpha release, and we don't support everything on that list yet. We support experiment tracking, training, and deployment, with plans to add data versioning and monitoring later. We very much welcome input on our roadmap from our early users. +Matcha is still in alpha release, and we don't support everything on that list yet. We support experiment tracking, training, data version control and deployment, with plans to add monitoring later. We very much welcome input on our roadmap from our early users. ## 👏 Contributing diff --git a/docs/index.md b/docs/index.md index 0991f4ad..10716684 100644 --- a/docs/index.md +++ b/docs/index.md @@ -40,7 +40,7 @@ We've put a lot of thought into what our users — data scientists, ML engineers * Somewhere to deploy and serve models in a way that scales with your application needs. * The ability to monitor models for things like drift and bias. -Matcha is still in alpha release, and we don't support everything on that list yet. We support experiment tracking, training, and deployment, with plans to add data versioning and monitoring later. We very much welcome input on our roadmap from our early users. +Matcha is still in alpha release, and we don't support everything on that list yet. We support experiment tracking, training, data versioning and deployment, with plans to add monitoring later. We very much welcome input on our roadmap from our early users. # Who maintains Matcha? diff --git a/docs/resource-stacks.md b/docs/resource-stacks.md index 64e1618c..a8f9bb21 100644 --- a/docs/resource-stacks.md +++ b/docs/resource-stacks.md @@ -49,4 +49,49 @@ $ matcha stack set llm If no stack is set Matcha will use the 'default' stack. + +## Custom stacks + +Need an additional module or to remove one? The custom stack is for you. + +You can start by adding a module to your stack as follows: + +```bash +$ matcha stack add experiment_tracker mlflow +``` + +```bash +$ matcha stack add +``` + +If you want to adapt one of the stacks above you can do this by first setting your stack: + +```bash +$ matcha stack set llm +``` + +then adding/removing a module from the stack + +```bash +$ matcha stack remove orchestrator +``` + +and then Matcha provision the specified resources: + +```bash +$ matcha provision +``` + +### Available Modules + +| Module Type | Flavors | Example Usage (add) | Example Usage (remove) | +|----------------------|---------|----------------------------------------------|---------------------------------------| +| orchestrator | zenml | `matcha stack add orchestrator zenml` | `matcha stack remove orchestrator` | +| experiment_tracker | mlflow | `matcha stack add experiment_tracker mflow` | `matcha stack remove orchestrator` | +| data_version_control | dvc | `matcha stack add data_version_control mflow`| `matcha stack remove data_version_control`| +| deployer | seldon | `matcha stack add deployer seldon` | `matcha stack remove deployer` | +| vector_database | chroma | `matcha stack add vector_database chroma` | `matcha stack remove vector_database` | + +> Note: If you have existing provisioned resources, you must destroy them before adding a new module. + See the [API documentation](references.md) for more information. From 56fada35b397b1fae5b24034809c56f28cb84ae5 Mon Sep 17 00:00:00 2001 From: Chris Date: Mon, 4 Sep 2023 13:40:33 +0100 Subject: [PATCH 2/3] Add stack list documentation --- docs/resource-stacks.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/resource-stacks.md b/docs/resource-stacks.md index a8f9bb21..1e235042 100644 --- a/docs/resource-stacks.md +++ b/docs/resource-stacks.md @@ -49,6 +49,24 @@ $ matcha stack set llm If no stack is set Matcha will use the 'default' stack. +### See what is in your current stack + +To list the modules that are in your current stack run: + +```bash +$ matcha stack list + + +The current Matcha stack contains:: + +1. Azure Kubernetes Service (AKS): A kubernetes cluster +2. Azure Container Registry: A container registry for storing docker images +3. ZenServer: A zenml server required for remote orchestration and a storage container +4. MLflow: An experiment tracker backed by a storage container +5. Data Version Control: A storage container to hold data versions +6. Seldon Core: A framework for model deployment on top of a kubernetes cluster +7. Chroma: A vector database +``` ## Custom stacks @@ -92,6 +110,6 @@ $ matcha provision | deployer | seldon | `matcha stack add deployer seldon` | `matcha stack remove deployer` | | vector_database | chroma | `matcha stack add vector_database chroma` | `matcha stack remove vector_database` | -> Note: If you have existing provisioned resources, you must destroy them before adding a new module. +> Note: Every stack will automatically contain Azure Kubernetes Service, Azure Container Registry and a state storage bucket. These components cannot be removed as they are required for Matcha to run correctly. See the [API documentation](references.md) for more information. From 9e28ad4701dbc88627b87963fbfffb97ec22c6b3 Mon Sep 17 00:00:00 2001 From: Chris Date: Mon, 4 Sep 2023 14:48:46 +0100 Subject: [PATCH 3/3] Add examples to add and remove stack modules in core --- src/matcha_ml/core/core.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/matcha_ml/core/core.py b/src/matcha_ml/core/core.py index b04e3c16..5951937f 100644 --- a/src/matcha_ml/core/core.py +++ b/src/matcha_ml/core/core.py @@ -394,6 +394,9 @@ def _create_stack_component(stack_type: StackType) -> MatchaConfigComponent: def stack_add(module_type: str, module_flavor: str) -> None: """A function for adding a module by name to the stack. + Examples: + >>> stack_add(module_type='experiment_tracker', module_flavor='mlflow') + Args: module_type (str): The type of the module to add e.g. 'experiment_tracker'. module_flavor (str): The flavor of module to add e.g. 'mlflow'. @@ -432,6 +435,9 @@ def stack_add(module_type: str, module_flavor: str) -> None: def stack_remove(module_type: str) -> None: """A function for removing a module by name in the stack. + Examples: + >>> stack_remove(module_type='experiment_tracker') + Args: module_type (str): The name of the module to remove.