-
Notifications
You must be signed in to change notification settings - Fork 1
Description
User Story
As a QType developer debugging data flows, I want to see how many rows each SQLSource step emits so that I can identify where data filtering occurs or understand query performance.
Problem/Motivation
When executing flows with SQLSource steps, there is no indication of how many rows each query returns. This makes it difficult to:
- Debug why downstream steps receive no data
- Understand if SQL queries are returning expected result sizes
- Identify performance bottlenecks in multi-step data pipelines
Proposed Solution
Log row counts after each SQLSource execution:
Executing SQL query with params: {...}
Emitted 1000 rows from SQL query
This information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is alreaumeThis information is already available in the code (line 100 in sql_source_executor.py shows len(df)) but nThis information is already available in the code (line 100 in sql_soith multiple SQLSource steps. Currently requires examining telemetry or adding debug steps.