feat(scheduler): add internal scheduler for self-hosted environments

[majiayu000]

Summary

• Add built-in internal scheduler for self-hosted environments
• Scheduler polls /api/schedules/execute to trigger scheduled workflows
• Enabled by default in docker-compose.prod.yml for self-hosted setups
• Configurable via ENABLE_INTERNAL_SCHEDULER, CRON_SECRET, and INTERNAL_SCHEDULER_INTERVAL_MS environment variables

Test plan

• Added unit tests for internal scheduler
• All tests pass (4 tests)
• Manual testing with Docker deployment

Fixes #1870

[vercel]

@majiayu000 is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR adds a built-in internal scheduler for self-hosted environments that periodically polls /api/schedules/execute to trigger scheduled workflows, solving the issue where schedule triggers don't work without external cron services like Vercel Cron.

Key Changes

• Internal Scheduler Module: New internal-scheduler.ts implements polling logic with configurable interval (default 60s), proper error handling, and graceful shutdown
• App Integration: Scheduler initializes during app startup via instrumentation-node.ts register hook
• Configuration: Three new environment variables added: ENABLE_INTERNAL_SCHEDULER, CRON_SECRET, and INTERNAL_SCHEDULER_INTERVAL_MS
• Docker Setup: Enabled by default in docker-compose.prod.yml with sensible defaults
• Test Coverage: Comprehensive unit tests for scheduler functionality

Implementation Details

The scheduler uses a simple polling mechanism with safeguards against concurrent executions. It authenticates using the existing CRON_SECRET via Bearer token, integrating seamlessly with the existing /api/schedules/execute endpoint that already supports both Trigger.dev and direct execution modes.

Issues Found

• Critical: docker-compose.prod.yml is missing required INTERNAL_API_SECRET environment variable, which will prevent the app from starting
• Style: Environment variable types should use z.boolean() and z.number() instead of z.string() for consistency with codebase patterns
• Minor: Signal handlers could accumulate if initializeInternalScheduler() is called multiple times (hot reload scenarios)

Confidence Score: 3/5

• This PR has a critical configuration issue that will prevent Docker deployments from starting, but the core implementation is solid
• The scheduler implementation itself is well-designed with proper error handling, tests, and graceful shutdown. However, the missing INTERNAL_API_SECRET in docker-compose.prod.yml is a critical issue that will cause immediate failures for users following the self-hosted Docker setup. The score reflects this deployment blocker rather than the quality of the scheduler code itself.
• docker-compose.prod.yml requires the missing INTERNAL_API_SECRET environment variable to be added before merge

Important Files Changed

Filename	Overview
apps/sim/lib/scheduler/internal-scheduler.ts	Core scheduler implementation with proper error handling and graceful shutdown. Implementation is solid but has a potential race condition with multiple SIGTERM/SIGINT handlers.
apps/sim/lib/core/config/env.ts	Added environment variables for scheduler configuration. Type should be z.boolean() instead of z.string() for consistency with other boolean flags.
docker-compose.prod.yml	Added scheduler configuration but missing required INTERNAL_API_SECRET environment variable which is needed for the app to start.

Sequence Diagram

sequenceDiagram
    participant App as Sim Application
    participant Scheduler as Internal Scheduler
    participant API as /api/schedules/execute
    participant DB as Database
    participant Workflow as Workflow Execution

    Note over App: App Startup (instrumentation-node.ts)
    App->>Scheduler: initializeInternalScheduler()
    
    alt ENABLE_INTERNAL_SCHEDULER !== 'true'
        Scheduler-->>App: Skip initialization
    else CRON_SECRET not configured
        Scheduler-->>App: Log warning, skip
    else Configuration valid
        Scheduler->>Scheduler: startInternalScheduler()
        Note over Scheduler: Set interval timer (default: 60s)
        Scheduler->>Scheduler: pollSchedules() immediately
    end

    loop Every INTERNAL_SCHEDULER_INTERVAL_MS
        Scheduler->>Scheduler: Check if previous poll running
        alt Previous poll still running
            Scheduler->>Scheduler: Skip this cycle
        else Ready to poll
            Scheduler->>API: GET /api/schedules/execute<br/>Authorization: Bearer {CRON_SECRET}
            API->>API: verifyCronAuth()
            
            alt Auth fails
                API-->>Scheduler: 401 Unauthorized
                Scheduler->>Scheduler: Log error
            else Auth succeeds
                API->>DB: Query due schedules<br/>(nextRunAt <= now, status != disabled)
                DB-->>API: Return due schedules
                API->>DB: Update lastQueuedAt
                
                loop For each due schedule
                    alt Trigger.dev enabled
                        API->>Workflow: tasks.trigger('schedule-execution')
                    else Trigger.dev disabled
                        API->>Workflow: executeScheduleJob() directly
                    end
                end
                
                API-->>Scheduler: 200 OK {executedCount: N}
                Scheduler->>Scheduler: Log execution count
            end
        end
    end

    Note over App: Graceful Shutdown (SIGTERM/SIGINT)
    App->>Scheduler: Signal handler triggered
    Scheduler->>Scheduler: stopInternalScheduler()
    Scheduler->>Scheduler: clearInterval()

Loading

[greptile-apps]

Additional Comments (5)

1. apps/sim/lib/core/config/env.ts, line 128-129 (link)

   style: Use z.boolean() instead of z.string() for boolean environment variables

   For consistency with other boolean flags in this file (like DISABLE_REGISTRATION, BILLING_ENABLED, TRIGGER_DEV_ENABLED), use proper boolean type:

   _{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

2. apps/sim/lib/scheduler/internal-scheduler.ts, line 126-133 (link)

   style: Multiple signal handlers registered on every initialization

   If initializeInternalScheduler() is called multiple times (e.g., in hot reload or tests), this adds duplicate SIGTERM/SIGINT handlers. Consider checking if handlers already exist or using process.once():

3. apps/sim/lib/scheduler/internal-scheduler.ts, line 79 (link)

   style: Handle invalid INTERNAL_SCHEDULER_INTERVAL_MS values

   Number() returns NaN for invalid strings. Consider validating or using parseInt with error handling:

4. docker-compose.prod.yml, line 11-26 (link)

   logic: Missing required INTERNAL_API_SECRET environment variable

   The app requires INTERNAL_API_SECRET (min 32 chars) to start, as defined in apps/sim/lib/core/config/env.ts:28. Add it after ENCRYPTION_KEY.

5. apps/sim/lib/scheduler/internal-scheduler.ts, line 106-107 (link)

   style: If ENABLE_INTERNAL_SCHEDULER is changed to z.boolean() in env.ts (as suggested in another comment), update comparison from === 'true' to === true

_{6 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}