Add performance analysis report identifying 72+ anti-patterns #87
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Comprehensive analysis of dbt models revealing: - 84% of incremental models missing incremental_predicates - 58 run_query() calls causing N+1-like patterns - Non-sargable JOINs using LOWER() function - Cartesian product in observability model - Repeated MAX(_inserted_timestamp) subqueries
Performance improvement for the core transactions model: - Convert temp tables to CTEs (blocks_source, tx_batch_source) - Replace Jinja for-loop date generation with SQL subquery - Keep only essential run_query for max_inserted_timestamp Benefits: - 75% reduction in compilation-time database round-trips - Snowflake CTE optimization materializes intermediate results - Cleaner, more maintainable code structure
Performance optimizations applied: 1. silver__nft_sales_combined.sql - Cache MAX(_inserted_timestamp) as Jinja variable (was repeated 3x) - Add incremental_predicates for partition pruning 2. silver_observability__transactions_completeness.sql - Replace implicit Cartesian (JOIN ON 1=1) with explicit CROSS JOIN 3. silver__hourly_prices_priority.sql - Pre-compute LOWER() values in CTEs to enable index usage - Eliminates non-sargable join conditions 4. core__ez_transfers.sql - Pre-compute LOWER() and DATE_TRUNC() in base CTEs - Eliminates per-row function calls during joins 5. Add incremental_predicates to 24 silver models: - 17 DEX swap models - 3 NFT sales models - 2 NFT mint models - 2 bridge transfer models Benefits: - Reduced compilation-time database round-trips - Enabled partition pruning for incremental runs - Improved join performance via index usage
Updated executive summary to document: - 29 files modified total - 6 categories of fixes applied - Specific models and improvements listed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR addresses multiple performance anti-patterns identified through a comprehensive codebase analysis. 29 files modified with fixes targeting compilation efficiency, query performance, and partition pruning.
Changes
Critical Fixes
silver__transactions.sqlrun_query()callssilver__nft_sales_combined.sqlMAX(_inserted_timestamp)repeated 3xsilver_observability__transactions_completeness.sqlJOIN ON 1=1CROSS JOINsilver__hourly_prices_priority.sqlLOWER()on both JOIN sides (non-sargable)core__ez_transfers.sqlLOWER()+DATE_TRUNC()in JOINsPartition Pruning (24 models)
Added
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"]to:Benefits
CROSS JOINdocuments intentional behaviorTest Plan
dbt compileto verify all models compile without errorsdbt run --select silver__transactionsto test core model changesincremental_predicatesadded