Skip to content

Comments

feat: add DuckLake datasource#121

Merged
anoop-narang merged 3 commits intomainfrom
feat/ducklake-datasource
Feb 13, 2026
Merged

feat: add DuckLake datasource#121
anoop-narang merged 3 commits intomainfrom
feat/ducklake-datasource

Conversation

@anoop-narang
Copy link
Contributor

@anoop-narang anoop-narang commented Feb 13, 2026

For #120

Summary

  • Add DuckLake as a new datasource using the datafusion-ducklake crate (v0.0.6)
  • DuckLake stores metadata in a Postgres catalog database and data as Parquet files on S3
  • Implements table discovery via CatalogProvider/SchemaProvider traversal and data fetch via DataFusion SessionContext
  • Credentials passed as a single JSON blob (catalog_db_password + optional S3 keys)

Usage

curl --location 'http://localhost:3000/connections' \
--header 'Content-Type: application/json' \
--data-raw '{
  "name": "my_ducklake",
  "source_type": "ducklake",
  "config": {
    "catalog_url": "postgresql://<user>@<host>:<port>/<database>?sslmode=require",
    "s3_endpoint": "https://s3.<region>.amazonaws.com",
    "s3_region": "<region>",
    "credentials_json": "{\"catalog_db_password\": \"<password>\", \"s3_access_key_id\": \"<access_key>\", \"s3_secret_access_key\": \"<secret_key>\"}"
  }
}'

Add DuckLake as a new datasource using the datafusion-ducklake library.
DuckLake stores metadata in a catalog database (Postgres) and data as
Parquet files on S3 or local storage.

- Add Source::Ducklake variant with catalog_url, s3_endpoint, s3_region
- Implement discover_tables via CatalogProvider/SchemaProvider traversal
- Implement fetch_table via DataFusion SessionContext with S3 object store
- Extract S3 bucket from catalog data_path for correct store registration
- Credentials passed as JSON blob (catalog_db_password + optional S3 keys)
@anoop-narang anoop-narang merged commit 2a3f98a into main Feb 13, 2026
8 checks passed
@anoop-narang anoop-narang deleted the feat/ducklake-datasource branch February 13, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant