Proposal: Integrated Multi-Repository and Layered Provisioning Architecture for azd

[spboyer]

Summary

This proposal outlines a comprehensive plan to enable first-class multi-repository (multi-repo) application scenarios in Azure Developer CLI (azd), fully integrating support for layered provisioning, dependency management, and enterprise-scale infrastructure patterns. It addresses key challenges from monolith-to-multirepo migration, cross-repository orchestration, and layered infrastructure deployment, referencing all major related issues.

---

Background & Motivation

azd currently uses a monorepo structure, where all services and infrastructure definitions live in a single repository. There is growing demand, especially from enterprise teams, to enable each service and/or infrastructure layer to reside in separate repositories—enabling team autonomy, better ownership, independent versioning, improved scalability, and alignment with platform engineering practices.

Key related issues:

• Support multi-repo application scenarios (#2372)
• Feature request: Support multiple infrastructure deployments in AZD provision cmd. (#4755)
• Create layer deployment orchestration engine (#5294)
• Multi environment support (#383)
• Create precedence resolution logic (#5322)

---

Comprehensive Multi-Repo + Layered Provisioning Plan

1. Extend azure.yaml Schema

• Add first-class repository support for both services and infra.layers:
  • Allow each service and each infra layer to specify an external git repository URL, branch/ref, and credentials.
  • Allow a mix of local and remote components.
• Example:

  infra:
    layers:
      - name: networking
        repository: 
          url: https://github.com/myorg/infra-networking
          ref: main
          credentials: $PLATFORM_PAT
        path: infra
        provider: bicep
      - name:  application
        path: infra
        provider: bicep
        dependsOn:
          - networking
  services:
    api: 
      repository:
        url: https://github.com/myorg/api-service
        ref: main
        credentials: $SERVICE_PAT
      project: ./src
      host: containerapp
      dependsOn:
        - application

2. Repository Manager & Workspace

• Implement a RepositoryManager to clone/fetch repos, manage credentials, cache, and track refs.
• Standardize workspace layout under .azure/workspace for clean, conflict-free source isolation.

3. Layered Orchestration & Dependency Resolution

• Enhance layer orchestration to support cross-repo dependencies:
  • Design a dependency graph (layers/services → layers/services).
  • Use a dependsOn: key for both infra layers and services.
  • Ensure provision, destroy, and CI/CD flows respect dependency order.

4. Provisioning Engine Enhancements

• Refactor provisioning logic to iterate provision/destroy by layer/service in topological order:
  • Support partial (layer/service) operations and promote re-entrancy.
  • Tag deployments/resources with layer+repo metadata for traceability.

5. Cross-Repo CI/CD Orchestration

• Generate or recommend workflows that can trigger downstream repo workflows (e.g., GitHub Actions, Azure Pipelines).
• Example: trigger service deployment after successful layer provision.

6. Configuration, Precedence & Security

• Add configuration/secret management for repository credentials.
• Implement clear precedence when configuration is present both locally and in dependencies (see Create precedence resolution logic #5322).
• Document security best practices for credential isolation and access control.

7. User Experience & Migration

• Provide migration guides and tooling for moving from monorepo to multi-repo setups.
• Ensure backwards compatibility—monorepo and multi-repo models should co-exist.

8. Performance & Caching

• Add smart caching for repo clones & builds.
• Detect no-op scenarios for unchanged layers/services.

9. Testing and Rollout

• Provide real-world starter templates for multi-repo/layered apps.
• Target incremental delivery: alpha flag → beta → GA.

---

Implementation Roadmap & Phasing

1. RepositoryManager POC: Clone, fetch, validate remote layers and services.
2. Schema validation: Extend and strictly validate azure. yaml.
3. Provisioning engine: Layered, cross-repo, dependency-aware orchestration.
4. CI/CD generation & docs: Multi-repo workflow blueprints and documentation.
5. User migration & feedback loop: Engage early adopters, iterate.

---

Out of Scope / Open Questions

• Support for non-Git backends
• Partial layer overrides/patching mechanics

---

References to Related Issues

• #2372: Support multi-repo application scenarios
• #4755: Support multiple infrastructure deployments in AZD provision cmd.
• #5294: Create layer deployment orchestration engine
• #383: Multi environment support
• #5322: Create precedence resolution logic

---

[vhvb1989]

Here's an existing multi-repo project managed with azd (azd + hooks)
https://github.com/Azure/GPT-RAG/blob/main/azure.yaml

The pattern used in that project is:

• Use one based monorepo for the infrastructure and config to other repos.
• Define a manifest file inside the /infra folder. It calls component to each other repository. Note that is used the release number to lock what version to use.
• Use a pre-deploy hook to orchestrate getting each linked repo and running azd deploy to each.
• Add a readme to each linked repo within the infra folder to describe the steps required to deploy the project by starting with the based repo.

This could be considered as prior art 🤠. Some time ago I helped to make this work using hooks and it has been evolving to use releases and to combine azd and az for what they need.