-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem
When a QType flow executes successfully but produces no output, there is no visibility into which step in the pipeline dropped all the data. This makes debugging complex flows extremely difficult.
Reproduction
Given a flow with multiple steps that filter or transform data:
flows:
- type: Flow
id: my_flow
interface:
type: Complete
inputs:
- input_data
outputs:
- result
steps:
- type: SQLSource
id: query_data
query: "SELECT * FROM table WHERE condition"
outputs:
- row_id
- type: Collect
id: collect_ids
inputs:
- row_id
outputs:
- all_ids
- type: SQLSource
id: query_details
query: "SELECT * FROM details WHERE id IN (:ids)"
inputs:
- all_i - all_i - all_i - all_i - all_i - all_i - all_i - all_i - all_i - all_i - alle d - all_i - all_i - all_i - all_i - all_i - all_i - all_i - all_i - all_i - all_i - alle d hav - al26-02-05 16:25:59,680 - qtype.commands.run - INFO - ✅ Flow execution complete cce 26-02 - a680 - aan - all_i ow wi - all_i` -ti - a caused y result.
## Impact
- Users must add debug step- Users must add debug step- Users must add debugDiffic- Users must ad i- Users must add query, p- Users must add or C- Users must add debuglarly problematic - Users must add tiple fan-out/fan-i- Users must add das- Udebugging time from minutes to hours for complex flows- Users must add debug step- Users must add debug step- User needs to know if the query executed but found nothing, or if parameters were wrong
2. **Collect step after zero-row SQLSource**: User needs visibility that upstream step emitted nothing
3. **Multiple filtering steps**: User needs to see where in the pipeline data was filtered out
## Proposed Solution
Add row count logging for key step types:
- SQLSource: Log number of rows emitted after query execution
- Collect: Log number of input items and resulting list size
- Filter/Map steps: Log input vs output counts
Could be enabled with existing `--verbose` flag or always shown when output is empty.Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request