[WIP] Feat: update existing resources instead of wholesale recreation

[jhcipar]

Overview: update existing resources instead of wholesale recreation

Objective: Enable in-place resource updates instead of costly redeploys when configuration changes are detected.

Previously, any configuration change would trigger a complete resource teardown and redeploy cycle.
Solution Overview
This PR introduces anupdate system that:

Detects configuration changes using content-based hashing
Updates resources in-place via platform APIs when possible
Handles complex updates that may require both template and endpoint modifications

Key Changes

1. Durable Resource Identity System

resource_id: Now represents a logical, human-readable identifier (ResourceType_name)

Provides stable identity across configuration changes
Enables resource reuse and update tracking
Replaces the previous hash-based approach for resource identification

resource_hash: Content-based hash for change detection

Built from _hashed_fields - only mutable configuration parameters
Excludes platform state (IDs, deployment metadata) to focus on user-controllable config
Triggers update flow when hash changes between runs

2. Update Logic

# New update path in ResourceManager.get_or_deploy_resource()
if existing.resource_hash != config.resource_hash:
    # Identify specific changed fields
    for field in existing.__class__._hashed_fields:
        if getattr(existing, field) != getattr(config, field):
            config.fields_to_update.add(field)
    
    # Sync deployment state and update in-place
    await config.sync_config_with_deployed_resource(existing)
    deployed_resource = await config.update()

3. Platform Integration

New GraphQL operations: update_endpoint() and update_template() mutations
Granular updates: System determines whether template, endpoint, or both need updating
State preservation: Maintains platform IDs and deployment metadata across updates

4. Enhanced Resource Model

_hashed_fields: Class-level definition of configuration fields that trigger updates
fields_to_update: Runtime tracking of specific changes to optimize update operations
sync_config_with_deployed_resource(): Transfers deployment state between resource instances

Bug Fixes

GPU configuration persistence: Fixed issue where gpuIds wasn't being properly stored in pickled resource state
Template ID tracking: Ensures template relationships are maintained through update cycles

Logic flow for resource update/creation

flowchart TD
    A[get_or_deploy_resource called] --> B[Acquire resource lock]
    B --> C{Resource exists?}
    
    C -->|No| D[Deploy new resource]
    D --> E[Add to manager & save]
    E --> F[Return deployed resource]
    
    C -->|Yes| G{Is resource deployed?}
    G -->|No| H[Remove invalid resource]
    H --> I[Deploy new resource]
    I --> J[Add to manager & save]
    J --> K[Return deployed resource]
    
    G -->|Yes| L{resource_hash changed?}
    L -->|No| M[Resource unchanged]
    M --> N[Return existing resource]
    
    L -->|Yes| O[Config change detected]
    O --> P[Compare _hashed_fields]
    P --> Q[Identify changed fields]
    Q --> R[Add to fields_to_update set]
    R --> S[sync_config_with_deployed_resource]
    S --> T[Call resource.update]
    
    T --> U{Pod template needs update?}
    U -->|Yes| V[Update template via GraphQL]
    V --> W{Template-only changes?}
    W -->|Yes| X[Return updated resource]
    W -->|No| Y[Update endpoint via GraphQL]
    
    U -->|No| Y
    Y --> Z[Remove old resource]
    Z --> AA[Add updated resource]
    AA --> BB[Return updated resource]
    

Loading

In the future, we'll have to integrate with durable Tetra state on the server side.