feat: Hot reload of confd.toml main configuration file

[abtreece]

Summary

Enable reloading confd's own configuration file without restarting the process, allowing dynamic reconfiguration.

Motivation

Currently, changing confd configuration requires:

1. Edit confd.toml
2. Restart confd process
3. Brief gap in configuration management

For long-running confd instances, this creates unnecessary downtime and operational complexity.

Proposed Implementation

Reload Trigger

# Via signal
kill -SIGHUP $(pidof confd)

# Via admin endpoint (if metrics server enabled)
curl -X POST http://localhost:9100/reload

# Via file watch (optional)
confd --watch-config

Reloadable Settings

Setting	Reloadable	Notes
interval	✅ Yes	Takes effect next cycle
log_level	✅ Yes	Immediate
template resources (conf.d/)	✅ Yes	Add/remove/modify templates
prefix	✅ Yes	Re-fetch keys
backend type	❌ No	Requires restart
backend address	⚠️ Partial	Reconnects if changed
watch mode	❌ No	Requires restart

Reload Sequence

SIGHUP received
    │
    ▼
┌─────────────────────────┐
│ 1. Parse new config     │
│    (validate first)     │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 2. Diff with current    │
│    configuration        │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 3. Apply reloadable     │
│    changes              │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 4. Reload template      │
│    resources            │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 5. Log applied changes  │
└─────────────────────────┘

Template Resource Hot Reload

# Add new template resource
cp new-service.toml /etc/confd/conf.d/
kill -SIGHUP $(pidof confd)
# New template is picked up

# Modify existing template resource
vim /etc/confd/conf.d/nginx.toml
kill -SIGHUP $(pidof confd)
# Changes take effect

# Remove template resource
rm /etc/confd/conf.d/old-service.toml
kill -SIGHUP $(pidof confd)
# Template is unloaded, watches stopped

Configuration Validation

Before applying reload:

func (c *Config) ValidateReload(new *Config) error {
    // Check for incompatible changes
    if c.Backend != new.Backend {
        return errors.New("backend type cannot be changed via reload")
    }
    
    // Validate new template resources
    for _, r := range new.TemplateResources {
        if err := r.Validate(); err != nil {
            return fmt.Errorf("template %s: %w", r.Src, err)
        }
    }
    
    return nil
}

Automatic Config Watching

[confd]
# Watch confd.toml for changes and auto-reload
watch_config = true

# Debounce config file changes
config_reload_debounce = "2s"

Status Reporting

INFO: Received SIGHUP, reloading configuration
INFO: Configuration validated successfully
INFO: Applied changes:
INFO:   - log_level: info -> debug
INFO:   - interval: 600 -> 300
INFO:   - Added template resource: new-service.toml
INFO:   - Modified template resource: nginx.toml
INFO:   - Removed template resource: old-service.toml
INFO: Configuration reload complete

Failed Reload Handling

If reload fails:

1. Log detailed error message
2. Keep running with existing configuration
3. Set error metric/status for monitoring
4. Return non-zero from admin endpoint

ERROR: Configuration reload failed: template new-service.toml: src file not found
ERROR: Keeping existing configuration

Benefits

• Zero-downtime configuration changes
• Dynamic template resource management
• Log level changes for debugging
• Interval tuning without restart
• Reduced operational overhead

[abtreece]

🟡 Partially Addressed in PR #429

PR #429 implements template resource hot reload via SIGHUP, which addresses the core use case for zero-downtime updates. However, reloading the main confd.toml configuration file is not yet implemented.

✅ What Works Now

SIGHUP Signal Handling

kill -HUP $(pidof confd)
# or
systemctl reload confd

Template Resource Reload

# Add new template resource
cp new-service.toml /etc/confd/conf.d/
kill -HUP $(pidof confd)
# ✅ New template is picked up

# Modify existing template resource
vim /etc/confd/conf.d/nginx.toml
kill -HUP $(pidof confd)
# ✅ Changes take effect

# Modify template file
vim /etc/confd/templates/nginx.conf.tmpl
kill -HUP $(pidof confd)
# ✅ Template cache cleared, reloaded on next use

# Remove template resource
rm /etc/confd/conf.d/old-service.toml
kill -HUP $(pidof confd)
# ✅ Template unloaded, watches stopped

What Gets Reloaded:

• ✅ Template resources from conf.d/ directory
• ✅ Template files from templates/ directory
• ✅ Template cache is cleared
• ✅ Watches are restarted with new configuration
• ✅ Zero downtime during reload

Reload Sequence:

SIGHUP received
    │
    ▼
┌─────────────────────────┐
│ 1. Clear template cache │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 2. Stop all watches     │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 3. Reload template      │
│    resources from       │
│    conf.d/              │
└─────────────────────────┘
    │
    ▼
┌─────────────────────────┐
│ 4. Restart watches      │
└─────────────────────────┘

❌ What Doesn't Work Yet

Main Configuration Reload

# Modify main config
vim /etc/confd/confd.toml
kill -HUP $(pidof confd)
# ❌ Main config NOT reloaded, requires restart

Settings that still require restart:

• Backend type (backend = "etcd")
• Backend nodes/addresses
• Watch vs. interval mode
• Global settings (log level, prefix, etc.)
• Metrics address
• Timeout values

Features not yet implemented:

• ❌ HTTP admin endpoint (POST /reload)
• ❌ Automatic config file watching (--watch-config)
• ❌ Pre-validation before applying reload
• ❌ Diff reporting of config changes

Impact

90% of the use case is now solved:

• The most common operation (updating templates) works without restart
• Template resources can be added/removed dynamically
• This covers the majority of deployment scenarios

Remaining 10%:

• Changing backend configuration still requires restart
• Tuning global settings (timeouts, log level) requires restart

Recommendation

Since template resource reload is the primary use case and is now working, I suggest:

1. Keep this issue open to track main config reload
2. Update the title to clarify scope: "feat: Hot reload of confd.toml main configuration file"
3. Create new milestones:
   • ✅ Phase 1: Template resource reload (DONE in feat: add production service deployment features #429)
   • ⬜ Phase 2: Main config reload (REMAINING)

The template reload functionality is production-ready and solves the most common pain point.

Documentation

See the service deployment guide for SIGHUP usage:

• Service Deployment Guide - SIGHUP Reload
• systemd ExecReload=/bin/kill -HUP $MAINPID