feat(encryption) [2/N] Support encryption: Add Table Properties for Encryption Configuration #2030
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xanderbailey
force-pushed
the
xb/encryption_table_properties
branch
from
679eb2d to
15d00e7
Compare
This PR introduces table-level encryption properties to enable configuration of encryption settings for Iceberg tables. These properties lay the groundwork for future encryption implementation while maintaining compatibility with the Java implementation's property names and structure. Table-level encryption is a critical security feature in Apache Iceberg's Java implementation. To support encryption in iceberg-rust and ensure interoperability between Java and Rust implementations, we need to start by adding the configuration properties that control encryption behavior. This PR adds the property definitions and parsing logic without implementing the actual encryption, keeping the change focused and reviewable. **Modified:** `crates/iceberg/src/spec/table_properties.rs` Added encryption-related properties to the `TableProperties` struct: - `PROPERTY_ENCRYPTION_KEY_ID` (`"encryption.key-id"`) - Master key ID for encrypting data encryption keys - `PROPERTY_ENCRYPTION_DEK_LENGTH` (`"encryption.data-key-length"`) - Data encryption key length (default: 16 bytes) - `PROPERTY_ENCRYPTION_AAD_LENGTH` (`"encryption.aad-length"`) - AAD prefix length for GCM (default: 16 bytes) - `PROPERTY_ENCRYPTION_KMS_TYPE` (`"encryption.kms-type"`) - KMS type (e.g., "aws", "gcp", "azure") All `Option<T>` as encryption is optional: - `encryption_key_id: Option<String>` - `encryption_dek_length: Option<usize>` - `encryption_aad_length: Option<usize>` - `encryption_kms_type: Option<String>` Extended `TryFrom<&HashMap<String, String>>` implementation to parse encryption properties Property names match exactly with Java's implementation: - Java: `TableProperties.ENCRYPTION_TABLE_KEY` → Rust: `PROPERTY_ENCRYPTION_KEY_ID` - Java: `TableProperties.ENCRYPTION_DEK_LENGTH` → Rust: `PROPERTY_ENCRYPTION_DEK_LENGTH` - Java: `CatalogProperties.ENCRYPTION_KMS_TYPE` → Rust: `PROPERTY_ENCRYPTION_KMS_TYPE` **Note:** Java's `ENCRYPTION_KMS_IMPL` property (for custom KMS implementations via reflection) is intentionally not included since Rust doesn't support runtime reflection. KMS implementations will be selected based on the `encryption.kms-type` property with compiled-in implementations. Added comprehensive test coverage: 1. `test_table_properties_default`: Verifies encryption properties are None by default 2. `test_encryption_properties_valid`: Tests parsing all encryption properties with valid values 3. `test_encryption_properties_partial`: Tests partial encryption configuration 4. `test_encryption_properties_invalid_numeric`: Verifies invalid numeric values are handled gracefully (parsed as None) 5. `test_encryption_properties_with_other_properties`: Tests encryption properties alongside existing table properties All tests pass: ``` running 7 tests test spec::table_properties::tests::test_table_properties_default ... ok test spec::table_properties::tests::test_encryption_properties_partial ... ok test spec::table_properties::tests::test_encryption_properties_invalid_numeric ... ok test spec::table_properties::tests::test_encryption_properties_valid ... ok test spec::table_properties::tests::test_encryption_properties_with_other_properties ... ok test spec::table_properties::tests::test_table_properties_valid ... ok test spec::table_properties::tests::test_table_properties_invalid ... ok ``` 1. **Optional Fields**: All encryption properties are `Option<T>` since encryption is an optional feature 2. **Silent Failure for Invalid Numbers**: Invalid numeric values for `dek_length` and `aad_length` are parsed as None rather than failing, matching the pattern for optional properties 3. **No Validation**: This PR doesn't validate property values (e.g., valid key lengths), leaving that for the encryption implementation 4. **No Custom KMS**: Omitted `encryption.kms-impl` property since Rust lacks reflection - KMS type selection will use `encryption.kms-type` with a factory pattern 5. **Independent PR**: No dependencies on other encryption code, can be merged independently This PR is part of a series to implement encryption support: - ✅ PR 1: Core encryption primitives (AES-GCM operations) - ✅ PR 2: Table properties for encryption (this PR) - PR 3: Key management interfaces - PR 4: EncryptionManager implementation - PR 5: Native Parquet encryption support - PR 6: Integration with Table and FileIO
xanderbailey
force-pushed
the
xb/encryption_table_properties
branch
from
15d00e7 to
d090eb3
Compare
xanderbailey
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pat of #2034
This PR introduces table-level encryption properties to enable configuration of encryption settings for Iceberg tables. These properties lay the groundwork for future encryption implementation while maintaining compatibility with the Java implementation's property names and structure.
Table-level encryption is a critical security feature in Apache Iceberg's Java implementation. To support encryption in iceberg-rust and ensure interoperability between Java and Rust implementations, we need to start by adding the configuration properties that control encryption behavior. This PR adds the property definitions and parsing logic without implementing the actual encryption, keeping the change focused and reviewable.
Modified:
crates/iceberg/src/spec/table_properties.rsAdded encryption-related properties to the
TablePropertiesstruct:PROPERTY_ENCRYPTION_KEY_ID("encryption.key-id") - Master key ID for encrypting data encryption keysPROPERTY_ENCRYPTION_DEK_LENGTH("encryption.data-key-length") - Data encryption key length (default: 16 bytes)All
Option<T>as encryption is optional:encryption_key_id: Option<String>encryption_dek_length: Option<usize>Extended
TryFrom<&HashMap<String, String>>implementation to parse encryption propertiesProperty names match exactly with Java's implementation:
TableProperties.ENCRYPTION_TABLE_KEY→ Rust:PROPERTY_ENCRYPTION_KEY_IDTableProperties.ENCRYPTION_DEK_LENGTH→ Rust:PROPERTY_ENCRYPTION_DEK_LENGTHAdded comprehensive test coverage:
test_table_properties_default: Verifies encryption properties are None by defaulttest_encryption_properties_valid: Tests parsing all encryption properties with valid valuestest_encryption_properties_partial: Tests partial encryption configurationtest_encryption_properties_invalid_numeric: Verifies invalid numeric values are handled gracefully (parsed as None)test_encryption_properties_with_other_properties: Tests encryption properties alongside existing table propertiesOptional Fields: All encryption properties are
Option<T>since encryption is an optional featureSilent Failure for Invalid Numbers: Invalid numeric values for
dek_lengthare parsed asNonerather than failing, matching the pattern for optional propertiesNo Validation: This PR doesn't validate property values (e.g., valid key lengths), leaving that for the encryption implementation
Independent PR: No dependencies on other encryption code, can be merged independently
This PR is part of a series to implement encryption support:
Which issue does this PR close?
What changes are included in this PR?
Are these changes tested?