Skip to content

feat: Multi-stage template processing pipeline #339

@abtreece

Description

@abtreece

Summary

Enable chaining multiple processing stages for templates: render → transform → validate → output.

Motivation

Complex configurations often require:

  • Rendering a template, then formatting the output (e.g., JSON pretty-print)
  • Merging multiple templates into a single output
  • Post-processing with external tools (jsonnet, yq, envsubst)
  • Validation before final output

Currently, this requires external scripting or wrapper tools.

Proposed Implementation

Pipeline Configuration

[template]
src = "config.json.tmpl"
dest = "/etc/app/config.json"
keys = ["/app/*"]

# New: processing pipeline
[[template.pipeline]]
stage = "render"
# Default stage, renders Go template

[[template.pipeline]]
stage = "transform"
cmd = "jq -S '.'"  # Sort keys, pretty print

[[template.pipeline]]
stage = "validate"
cmd = "jq empty"   # Validate JSON syntax

[[template.pipeline]]
stage = "transform"
cmd = "yq -P"      # Convert to YAML (if needed)

Built-in Stages

[[template.pipeline]]
stage = "format"
format = "json"    # Pretty-print JSON
indent = 2

[[template.pipeline]]
stage = "format"
format = "yaml"    # Convert to YAML

[[template.pipeline]]
stage = "minify"
format = "json"    # Minify JSON output

[[template.pipeline]]
stage = "merge"
with = "/etc/confd/templates/defaults.json"
strategy = "deep"  # deep | shallow | replace

Template Merging

Combine multiple templates or static files:

[template]
dest = "/etc/app/config.json"

[[template.sources]]
src = "base.json.tmpl"
keys = ["/app/base/*"]

[[template.sources]]
src = "overrides.json.tmpl"
keys = ["/app/overrides/*"]

[template.merge]
strategy = "deep"  # Deep merge JSON objects

External Processors

[[template.pipeline]]
stage = "external"
cmd = "jsonnet"
args = ["-"]       # Read from stdin

Pipeline Flow

┌──────────┐    ┌───────────┐    ┌──────────┐    ┌────────┐
│  Render  │───▶│ Transform │───▶│ Validate │───▶│ Output │
└──────────┘    └───────────┘    └──────────┘    └────────┘
     │                │                │              │
     ▼                ▼                ▼              ▼
  Go template     jq/yq/etc      Check syntax    Write file

Error Handling

  • Pipeline stops on first error
  • Validation failures prevent output
  • Detailed error messages with stage info
  • Rollback support (keep previous file on failure)

Use Cases

  1. JSON config with schema validation
  2. YAML generation from JSON templates
  3. Nginx config with syntax check (nginx -t)
  4. Kubernetes manifests with kubectl --dry-run
  5. Merged environment-specific configs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions