Skip to content

No visibility into row counts emitted by SQLSource during execution #141

@loukratz-bv

Description

@loukratz-bv

User Story

As a QType developer debugging data flows, I want to see how many rows each SQLSource step emits so that I can identify where data filtering occurs or understand query performance.

Problem/Motivation

When executing flows with SQLSource steps, there is no indication of how many rows each query returns. This makes it difficult to:

  • Debug why downstream steps receive no data
  • Understand if SQL queries are returning expected result sizes
  • Identify performance bottlenecks in multi-step data pipelines

Proposed Solution

Log row counts after each SQLSource execution:

Executing SQL query with params: {...}
Emitted 1000 rows from SQL query

This information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is alreaumeThis information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is already available in the code (line 100 in sql_soith multiple SQLSource steps. Currently requires examining telemetry or adding debug steps.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions