Skip to content

Flow completes with 'no output' but no indication which step dropped the data #142

@loukratz-bv

Description

@loukratz-bv

Problem

When a QType flow executes successfully but produces no output, there is no visibility into which step in the pipeline dropped all the data. This makes debugging complex flows extremely difficult.

Reproduction

Given a flow with multiple steps that filter or transform data:

flows:
  - type: Flow
    id: my_flow
    interface:
      type: Complete
    inputs:
      - input_data
    outputs:
      - result
    
    steps:
      - type: SQLSource
        id: query_data
        query: "SELECT * FROM table WHERE condition"
        outputs:
          - row_id
      
      - type: Collect
        id: collect_ids
        inputs:
          - row_id
        outputs:
          - all_ids
      
      - type: SQLSource
        id: query_details
        query: "SELECT * FROM details WHERE id IN (:ids)"
        inputs:
          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - alle d          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - all_i          - alle d       hav          - al26-02-05 16:25:59,680 - qtype.commands.run - INFO - ✅ Flow execution complete    cce          26-02          - a680          - aan          - all_i ow           wi          - all_i`          -ti          - a caused         y result.

## Impact

- Users must add debug step- Users must add debug step- Users must add debugDiffic- Users must ad i- Users must add  query, p- Users must add  or C- Users must add debuglarly problematic - Users must add tiple fan-out/fan-i- Users must add das- Udebugging time from minutes to hours for complex flows- Users must add debug step- Users must add debug step- User needs to know if the query executed but found nothing, or if parameters were wrong
2. **Collect step after zero-row SQLSource**: User needs visibility that upstream step emitted nothing
3. **Multiple filtering steps**: User needs to see where in the pipeline data was filtered out

## Proposed Solution

Add row count logging for key step types:
- SQLSource: Log number of rows emitted after query execution
- Collect: Log number of input items and resulting list size
- Filter/Map steps: Log input vs output counts

Could be enabled with existing `--verbose` flag or always shown when output is empty.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions