Consolidate authentication settings into unified 'auth' configuration

[edgarrmondragon]

Background

PR #488 introduced org_auth_app_keys to support organization-specific GitHub App authentication. While this solves an important use case, it adds to the growing number of authentication-related settings, which now include:

• auth_token - Single GitHub token (string)
• additional_auth_tokens - Array of additional tokens
• auth_app_keys - Array of org-agnostic GitHub App keys
• org_auth_app_keys - Dictionary mapping organizations to arrays of app keys

Related discussion: #488 (comment)

Existing State

The current authentication configuration is fragmented across multiple top-level settings:

# Current structure
auth_token: "main-token"
additional_auth_tokens:
  - "token-x"
  - "token-y"
auth_app_keys:
  - ":app_id:;;-----BEGIN RSA PRIVATE KEY-----..."
org_auth_app_keys:
  org-a:
    - ":app_id_a:;;-----BEGIN RSA PRIVATE KEY-----..."
    - "token-a2"
  org-b:
    - ":app_id_b:;;-----BEGIN RSA PRIVATE KEY-----..."

Issues with current approach:

• Setting proliferation makes configuration harder to understand
• No clear hierarchy or relationship between settings
• Can complicate schema validation in downstream tools (e.g., meltano/hub#1266)

Proposed Improvement

Consolidate all authentication settings under a single auth configuration object:

# Proposed structure
auth:
  token: "main-token"
  additional:
    - "token-x"
    - "token-y"
    - ":app_id:;;-----BEGIN RSA PRIVATE KEY-----..."
  orgs:
    org-a:
      - ":app_id_a:;;-----BEGIN RSA PRIVATE KEY-----..."
      - "token-a2"
    org-b:
      - ":app_id_b:;;-----BEGIN RSA PRIVATE KEY-----..."

Mapping to Current Settings

• auth_token → auth.token (single string)
• additional_auth_tokens → auth.additional (array)
• auth_app_keys → auth.additional (array, merged with additional_auth_tokens)
• org_auth_app_keys → auth.orgs (dict of arrays)

Benefits:

• Clearer hierarchy: All authentication settings grouped logically
• Simpler schema: Single object type instead of multiple top-level settings
• Better UX: Easier to understand the relationship between default, org-specific, and additional tokens
• No functionality loss: Maintains array support for org-specific tokens and additional tokens
• Backward compatible migration path: Can support both old and new formats during deprecation period

Implementation Details

The new auth setting would have:

• token (string, optional): Primary token to use (maps to auth_token)
• additional (array of strings, optional): Additional tokens for fallback and rate limit rotation (combines additional_auth_tokens and auth_app_keys)
• orgs (object of arrays, optional): Maps organization names to arrays of tokens (maps to org_auth_app_keys)

Token format detection would remain automatic - the tap already detects whether a credential is a personal access token or GitHub App key based on format.

Deprecation Implementation

Phase 1: Introduction (v1.26.0 or v1.27.0)

1. Add new auth configuration setting
2. Maintain full backward compatibility with existing settings
3. Internal logic:
   • If auth is provided, use it
   • If old settings are provided, convert them internally to the new structure
   • If both are provided, merge them with auth taking precedence
4. Add deprecation warnings when old settings are used
5. Update documentation to recommend new auth format

Phase 2: Soft Deprecation (~6 months after Phase 1)

1. Increase visibility of deprecation warnings
2. Update all examples and documentation to use new format only
3. Add migration guide to help users convert configurations
4. Consider adding a CLI tool or script to automatically convert configs

Phase 3: Hard Deprecation (v2.0.0)

1. Remove support for old settings
2. Raise clear error messages if old settings are detected, pointing to migration guide
3. Clean up internal code that handled backward compatibility

Timeline

Given the current version is v1.25.0:

• Phase 1: Target v1.26.0 or v1.27.0 (next minor release after PR feat: Add ability to authenticate multiple orgs in one stream #488)
• Phase 2: ~6 months after Phase 1 release
• Phase 3: v2.0.0 (next major version, 12-18 months after Phase 1)

This timeline provides ample time for users to migrate while keeping the deprecation period reasonable.

Open Questions

1. Should we provide a migration utility/script to automatically convert old configs to new format?
2. Should deprecation warnings include the converted config snippet to make migration easier?
3. Do we need a migration validation mode to help users test their new configs before fully switching?