Optimization v2 #3
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced optional dependencies for Parquet and Arrow libraries in Cargo.toml. - Implemented `with_parquet` and `with_parquet_and_ttl` constructors in Cache for Parquet persistence. - Added event serialization and deserialization using Serde for cache events. - Created a new module `parquet_store` for handling Parquet file operations, including writing and reading cache items. - Implemented background writer for persisting cache events to Parquet file. - Added example demonstrating the usage of Parquet persistence feature.
…roved value mapping
- Removed Parquet persistence support from `parquet_store_v2.rs`. - Added new SQLite persistence module in `sqlite_store.rs` for efficient cache operations. - Created a background writer for persisting events to SQLite. - Implemented functions for initializing the database, reading cache items, and handling TTL updates. - Added examples for testing SQLite persistence in `test_persist.rs`. - Introduced an interactive TUI example in `tui_interactive.rs` to demonstrate cache operations with SQLite.
- Bump ratatui and crossterm versions to 0.29 in Cargo.toml - Update rusqlite version to 0.37 with bundled feature - Remove serde from persist feature dependencies - Refactor tui_interactive.rs for improved readability and consistency - Adjust database path default to current directory - Clean up whitespace and formatting throughout the file
…rsistence, events, and TTL
- Migrated from std::collections::HashMap to hashbrown::HashMap for better performance - Performance improvements: 20-25% faster GET operations, 17-36% faster list operations - All tests passing, 100% API compatible (drop-in replacement) - Added comprehensive performance report documenting the improvements - No breaking changes, external API remains unchanged
- Added detailed performance benchmark results with hashbrown - Included test environment specifications (WSL2, AMD Ryzen 9 7900, 20GB RAM) - Added real-world performance measurements for all operations - Documented test results (20/20 passing) - Highlighted performance improvements from hashbrown migration - Added key performance insights and memory usage details
- Updated `sqlite_store.rs` to improve code readability and performance. - Replaced `serde_json` with `Value::json_to_value` for better value handling. - Consolidated multiple `execute_batch` calls into single statements where applicable. - Added comprehensive benchmarks for cache operations, including insert, get, remove, and TTL features. - Introduced new benchmark groups for advanced features and persistence. - Documented benchmark usage and results in `benches/README.md`. - Improved performance measurement and reporting in benchmark results.
… and TTL management - Introduced `CacheItem` struct to store values with creation time and optional TTL. - Created `Cache` struct utilizing `IndexMap` to maintain insertion order and provide efficient access. - Added methods for inserting, retrieving, and removing items, including support for default TTL and event notifications. - Implemented batch operations for improved performance. - Included functionality for persisting cache items to a database with optional TTL. - Added cleanup method to remove expired items and maintain cache efficiency.
- Add inline hints to small methods for better compiler optimization - #[inline(always)] for very small methods (is_expired, len, is_empty, capacity, etc) - #[inline] for small methods (event senders, setters) - Pre-allocate capacity in collections - HashMap and Vec now pre-allocate with cache capacity in all constructors - Optimized cleanup_expired() with capacity pre-allocation (10% estimate) Performance improvements: - Get operations: 24-28% faster - Contains_key: 12-14% faster - List with end filter: 47% faster - TTL get with expired: 26% faster - LRU eviction: 17% faster - Value types: 9-18% faster All tests passing, significant performance gains across the board.
…penho e ordenação
- Added string pooling to reduce memory allocations and improve performance for repetitive keys. - Introduced SIMD-based fast filters for optimized prefix and suffix matching. - Implemented memory prefetch hints to enhance cache locality and reduce cache misses. - Optimized TTL handling by using integer timestamps instead of Duration for faster expiration checks. - Improved test reliability by generating unique database paths for SQLite tests and robust cleanup of temporary files. - Enhanced performance across various operations, achieving significant speed improvements in inserts, gets, and cleanup processes. - Updated documentation to reflect new features and optimizations.
- Updated README.md to include advanced performance optimizations such as SIMD filters, memory prefetch hints, string pooling, and TTL optimizations, highlighting performance gains of up to 48%. - Added detailed sections on advanced performance features, benchmark results, and technical optimizations. - Removed outdated benchmark_results.txt file as performance data is now integrated into README.md.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.