Skip to content

Comments

Add S3/ObjectStore write support for DuckLake catalogs#49

Merged
shefeek-jinnah merged 4 commits intomainfrom
shefeek/s3_write_support
Feb 13, 2026
Merged

Add S3/ObjectStore write support for DuckLake catalogs#49
shefeek-jinnah merged 4 commits intomainfrom
shefeek/s3_write_support

Conversation

@shefeek-jinnah
Copy link
Collaborator

Summary

  • Convert write pipeline from std::fs to object_store::put() with buffer approach (ArrowWriter<Vec<u8>>)
  • Replace PathBuf with String throughout the write config chain (catalog, schema, table)
  • DuckLakeTableWriter::new() now takes Arc<dyn ObjectStore> as second parameter
  • finish(), write_table(), append_table() are now async
  • DuckLakeInsertExec obtains ObjectStore from RuntimeEnv (same pattern as the read path)
  • Dropped sessions are safe — buffer is discarded without uploading

@shefeek-jinnah shefeek-jinnah merged commit 5f48cf2 into main Feb 13, 2026
3 checks passed
@shefeek-jinnah shefeek-jinnah deleted the shefeek/s3_write_support branch February 13, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant