fix: prevent memory exhaustion in loops with bounded iteration outputs

[aadamsx]

Summary

This PR addresses memory exhaustion issues that occur when running workflows with loops containing agent blocks that make many tool calls (e.g., MCP file operations).

Fixes #2525

Problem

Memory accumulated unbounded in two key areas during workflow execution:

1. allIterationOutputs in LoopScope - every loop iteration pushed results to this array with no limit
2. blockLogs in ExecutionContext - every block execution added logs with no pruning

This caused OOM crashes on systems with 64GB+ RAM during long-running workflow executions with loops.

Solution

Added memory management with configurable limits in two places:

Loop Orchestrator (apps/sim/executor/orchestrators/loop.ts)

• New addIterationOutputsWithMemoryLimit() method
• Limits stored iterations to MAX_STORED_ITERATION_OUTPUTS (default: 100)
• Monitors memory size with MAX_ITERATION_OUTPUTS_SIZE_BYTES (default: 50MB)
• Discards oldest iterations when limits exceeded
• Logs warning when truncation occurs

Block Executor (apps/sim/executor/execution/block-executor.ts)

• New addBlockLogWithMemoryLimit() method
• Limits stored logs to MAX_BLOCK_LOGS (default: 500)
• Monitors memory size with MAX_BLOCK_LOGS_SIZE_BYTES (default: 100MB)
• Periodic size checks every 50 logs to avoid frequent JSON serialization
• Logs warning when truncation occurs

New Constants (apps/sim/executor/constants.ts)

• MAX_STORED_ITERATION_OUTPUTS: 100
• MAX_ITERATION_OUTPUTS_SIZE_BYTES: 50MB
• MAX_BLOCK_LOGS: 500
• MAX_BLOCK_LOGS_SIZE_BYTES: 100MB

Trade-offs

• Final aggregated loop.results will contain only the most recent iterations (up to 100)
• Block logs in execution data will contain only the most recent logs (up to 500)
• Warnings are logged when truncation occurs, allowing users to see if limits were hit

Testing

• For loop: Execute a workflow with 200+ loop iterations, verify memory doesn't grow unbounded
• Agent in loop: Run a loop with agent blocks making 50+ tool calls per iteration, verify no OOM

Files Changed

• apps/sim/executor/constants.ts - Added new configurable limits
• apps/sim/executor/orchestrators/loop.ts - Added memory-bounded iteration storage
• apps/sim/executor/execution/block-executor.ts - Added memory-bounded log storage

[vercel]

@aadamsx is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

addresses memory exhaustion in loops by adding configurable limits for iteration outputs (100 iterations, 50MB) and block logs (500 logs, 100MB). The implementation adds memory-bounded storage with count and size limits that trigger truncation when exceeded.

Critical Issues Found:

• Array slicing bug: loop.ts:486 and block-executor.ts:676 use .slice(discardCount) which keeps the NEWEST elements instead of oldest, opposite of intended behavior
• Performance issue: loop.ts:497 calls JSON.stringify() on every iteration instead of periodically (unlike block-executor.ts which checks every 50 logs)
• Error handling bug: Both files return max limit on serialization errors, preventing cleanup and allowing unbounded growth

Trade-offs:

• Final loop.results will only contain most recent 100 iterations (acceptable for long-running workflows)
• Block logs limited to most recent 500 entries (sufficient for debugging recent execution)
• Warnings logged when truncation occurs for observability

Confidence Score: 1/5

• unsafe to merge - contains critical logic bugs that break core functionality
• the array slicing logic in both files is inverted (keeps newest instead of oldest), the performance optimization defeats itself by serializing on every loop iteration, and error handling prevents cleanup when it's most needed
• apps/sim/executor/orchestrators/loop.ts and apps/sim/executor/execution/block-executor.ts both need immediate fixes to array slicing logic, periodic size checks, and error handling

Important Files Changed

Filename	Overview
apps/sim/executor/constants.ts	added four well-documented memory limit constants with reasonable defaults
apps/sim/executor/orchestrators/loop.ts	added memory-bounded iteration storage but has critical bugs: array slicing logic keeps wrong elements, JSON.stringify runs every iteration causing performance issues, error handling prevents cleanup
apps/sim/executor/execution/block-executor.ts	added memory-bounded log storage with periodic size checks, but array slicing logic keeps wrong elements and error handling prevents cleanup

Sequence Diagram

sequenceDiagram
    participant LE as Loop Execution
    participant LO as LoopOrchestrator
    participant Scope as LoopScope
    participant BE as BlockExecutor
    participant Ctx as ExecutionContext
    
    Note over LE,Ctx: Loop Iteration Flow with Memory Management
    
    LE->>LO: evaluateLoopContinuation(ctx, loopId)
    LO->>Scope: collect currentIterationOutputs
    LO->>LO: addIterationOutputsWithMemoryLimit(scope, results, loopId)
    
    Note over LO: Check count limit (100)
    alt allIterationOutputs.length > MAX_STORED_ITERATION_OUTPUTS
        LO->>Scope: slice() - discard oldest iterations
        LO->>LO: logger.warn() - log truncation
    end
    
    Note over LO: Check memory size (50MB)
    LO->>LO: estimateObjectSize(allIterationOutputs)
    Note right of LO: JSON.stringify() runs EVERY iteration<br/>(performance issue)
    alt estimatedSize > MAX_ITERATION_OUTPUTS_SIZE_BYTES
        LO->>Scope: slice() - discard oldest half
        LO->>LO: logger.warn() - log truncation
    end
    
    LO->>Scope: currentIterationOutputs.clear()
    LO-->>LE: continuation result
    
    Note over LE,Ctx: Block Execution Flow with Memory Management
    
    LE->>BE: execute(ctx, node, block)
    BE->>BE: createBlockLog(ctx, node.id, block, node)
    BE->>BE: addBlockLogWithMemoryLimit(ctx, blockLog)
    BE->>Ctx: blockLogs.push(blockLog)
    
    Note over BE: Check count limit (500)
    alt blockLogs.length > MAX_BLOCK_LOGS
        BE->>Ctx: blockLogs = blockLogs.slice(discardCount)
        BE->>BE: logger.warn() - log truncation
    end
    
    Note over BE: Periodic size check (every 50 logs)
    alt blockLogs.length % 50 === 0
        BE->>BE: estimateBlockLogsSize(blockLogs)
        Note right of BE: JSON.stringify() every 50 logs
        alt estimatedSize > MAX_BLOCK_LOGS_SIZE_BYTES
            BE->>Ctx: blockLogs = blockLogs.slice(discardCount)
            BE->>BE: logger.warn() - log truncation
        end
    end
    
    BE->>BE: execute block handler
    BE-->>LE: normalized output

Loading

[greptile-apps]

Additional Comments (5)

1. apps/sim/executor/orchestrators/loop.ts, line 519-529 (link)

   logic: JSON.stringify() runs on EVERY iteration before size check, defeating the purpose of memory optimization. With 1000 iterations, this serializes potentially GBs of data 1000 times.

   Move size check to only run periodically (e.g., every 10 iterations) like block-executor.ts:

   Then in addIterationOutputsWithMemoryLimit at line 497:

   // Check memory size limit periodically (every 10 iterations to avoid frequent serialization)
   if (scope.allIterationOutputs.length % 10 === 0) {
     const estimatedSize = this.estimateObjectSize(scope.allIterationOutputs)
     if (estimatedSize > DEFAULTS.MAX_ITERATION_OUTPUTS_SIZE_BYTES) {

2. apps/sim/executor/execution/block-executor.ts, line 704-710 (link)

   logic: returns max limit on error, which prevents cleanup and allows unbounded growth if serialization consistently fails

3. apps/sim/executor/orchestrators/loop.ts, line 519-529 (link)

   logic: same issue - returning max limit prevents cleanup on serialization errors

4. apps/sim/executor/orchestrators/loop.ts, line 483-494 (link)

   logic: slicing from discardCount removes NEWEST iterations instead of oldest. The intent is to keep the most recent 100 iterations.

5. apps/sim/executor/execution/block-executor.ts, line 673-682 (link)

   logic: slicing from discardCount removes NEWEST logs instead of oldest

_{3 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}