Add LSM Tree

[bootjp]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @bootjp, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the storage layer by integrating a new LSM Tree-based MVCC store using CockroachDB's Pebble library. This change provides a more advanced and efficient mechanism for managing versioned key-value data, offering improved performance characteristics and robust data handling, including transactional mutations and snapshotting, which are critical for building scalable and fault-tolerant systems.

Highlights

• New LSM Tree Implementation: Introduced a new pebbleStore that implements the MVCCStore interface, leveraging CockroachDB's Pebble LSM tree for persistent storage. This provides a robust and performant foundation for versioned key-value data.
• MVCC Operations: The pebbleStore supports core MVCC operations including GetAt, PutAt, DeleteAt, ScanAt, PutWithTTLAt, ExpireAt, and ApplyMutations, all respecting a given timestamp for consistency.
• Key and Value Encoding: Implemented a specific key encoding scheme (UserKey + Inverted Timestamp) to facilitate efficient MVCC lookups and range scans. Values are encoded using gob to store the actual data, tombstone status, and expiration timestamp.
• Snapshot and Restore Functionality: Added comprehensive snapshotting and restoration capabilities, allowing the entire state of the Pebble store to be serialized to and deserialized from an io.ReadWriter, crucial for features like Raft state machine replication.
• Dependency Updates: Updated go.mod and go.sum to include github.com/cockroachdb/pebble and its various indirect dependencies, such as github.com/DataDog/zstd, github.com/prometheus/client_golang, and others.
• Unit Tests: Included a new test file store/lsm_store_test.go with unit tests covering basic CRUD operations, time-travel queries, range scans, and the snapshot/restore mechanism for the pebbleStore.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces an LSM-Tree-based MVCC store using PebbleDB. The implementation is comprehensive, covering basic MVCC operations, transactions, and snapshotting. I've identified a few critical issues related to data consistency, error handling, and a bug in the scan implementation that could lead to incorrect behavior or performance degradation. Additionally, there are opportunities for performance improvements and completing some stubbed-out functionality. Overall, a solid foundation for a persistent MVCC store.