Skip to content

Add DuckDB integration for pipeline.dataset() #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nicosuave merged 1 commit into from
Jan 11, 2026
Merged

Add DuckDB integration for pipeline.dataset() #2

nicosuave merged 1 commit into from
Jan 11, 2026
+1,628 −802

Conversation

@nicosuave
Copy link
Member

@nicosuave nicosuave commented Jan 11, 2026

Summary

  • Adds WithSqlClient and SupportsOpenTables interfaces to IcebergRestClient
  • Implements IcebergSqlClient class for DuckDB views via iceberg_scan()
  • Enables querying loaded Iceberg data via pipeline.dataset()
  • Updates README with DuckDB documentation and new features

Usage

# After loading data
dataset = pipeline.dataset()

# Access as dataframe
df = dataset["events"].df()

# Run SQL queries
result = dataset.query("SELECT * FROM events WHERE value > 50").fetchall()

# Get Arrow table
arrow_table = dataset["events"].arrow()

Changes

  • src/dlt_iceberg/sql_client.py - New DuckDB SQL client
  • src/dlt_iceberg/destination_client.py - Added interface implementations
  • tests/test_dataset.py - Comprehensive tests
  • README.md - Updated with new features
  • Added duckdb as dependency

@nicosuave
Add DuckDB integration for pipeline.dataset()
d29a973 
- Implement WithSqlClient interface for IcebergRestClient
- Implement SupportsOpenTables interface for table access
- Add IcebergSqlClient class for DuckDB views via iceberg_scan()
- Add duckdb as dependency
- Add comprehensive tests for dataset integration
- Update README with new features and DuckDB documentation
@nicosuave nicosuave merged commit a618d20 into main Jan 11, 2026
1 check passed
@nicosuave nicosuave deleted the feature/duckdb-dataset branch January 11, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nicosuave