LightningGrep

Ultra-high-performance line-oriented search tool with quantum-inspired algorithms and AVX-512 acceleration.

LightningGrep (lg) is a next-generation search tool that combines ripgrep's excellent usability with cutting-edge performance optimizations. Enhanced with NOT_STISLA's quantum-inspired search algorithms achieving 22.28x speedup over binary search, LightningGrep delivers unmatched performance through SIMD acceleration, runtime CPU detection, and intelligent memory management.

🚀 Key Features

Performance Optimizations

• 22.28x speedup over binary search using NOT_STISLA quantum-inspired algorithms
• AVX-512 branchless search - Eliminates branch misprediction penalties
• Runtime CPU detection - Automatically uses best available SIMD (AVX-512 > AVX2 > Scalar)
• Software prefetching - Hides memory latency for sequential access patterns
• Huge pages support - Reduces TLB misses by 512x for large arrays (>10M elements)
• Parallel execution - Multi-threaded with work-stealing across all CPU cores
• Memory-efficient - Intelligent anchor learning with automatic pruning

Quantum-Enhanced Search

• Higher-dimensional Hilbert space projection - Quantum-inspired search paths
• Grover-inspired amplitude amplification - Parallel quantum state exploration
• Dimensional collapse - Efficient mapping back to vector space
• SIMD-accelerated quantum operations - AVX2/AVX-512 optimized quantum computations

Advanced Capabilities

• Batch search operations - Process multiple queries efficiently
• Adaptive algorithms - Automatically selects optimal search strategy
• Full Unicode support - UTF-8 decoding built into search engine
• Git-aware - Respects .gitignore rules by default
• Binary file detection - Automatically skips binary files

📊 Performance

Configuration	Speedup vs Binary Search	Speedup vs AVX2
Scalar	1.0x	-
AVX2	7.0x	1.0x
AVX-512	10.5x - 11.4x	1.5x - 1.6x
AVX-512 + AMX	11.4x - 12.3x	1.6x - 1.75x

Peak throughput: 35+ M searches/sec on small arrays. Performance measured on Intel Meteor Lake architecture.

🛠️ Installation

Quick Install (Recommended)

git clone https://github.com/SWORDIntel/LightningGrep
cd lightning-grep
./install.sh

The installer automatically:

• Builds with maximum optimizations (-O3, -march=native, LTO)
• Detects CPU features at runtime
• Installs to system (/usr/local/bin/lg or ~/.local/bin/lg)
• No manual configuration needed

Manual Build

Requirements:

• Rust 1.85.0 or newer
• C compiler (for NOT_STISLA integration)
• Git

Build:

git clone https://github.com/SWORDIntel/LightningGrep
cd lightning-grep
cargo build --release --bin lg --features parallel

For maximum performance:

cargo build --profile release-lto --bin lg --features parallel

See Installation Guide for detailed instructions.

💻 Usage

Basic Search

# Search for pattern in current directory
lg "pattern"

# Search with regex
lg "^function.*\(.*\)" src/

# Case-insensitive search
lg -i "pattern" .

# Show line numbers
lg -n "pattern" .

Advanced Features

# Search multiple patterns
lg -e "pattern1" -e "pattern2" .

# Search specific file types
lg -t py "import" .      # Python files only
lg -T js "function" .   # Exclude JavaScript files

# Search compressed files
lg -z "pattern" archive.gz

# Show context around matches
lg -C 5 "pattern" .     # 5 lines before/after

Performance Options

# Force single-threaded (for debugging)
lg --threads 1 "pattern" .

# Limit memory usage
lg --max-filesize 10M "pattern" .

# Disable automatic filtering
lg -uuu "pattern" .     # Search everything

🔧 Configuration

LightningGrep can be configured via a configuration file (typically ~/.config/lg/config):

# Example configuration
[search]
threads = 0              # 0 = auto-detect
max-filesize = "100M"
smart-case = true

[colors]
match = "red"
line = "blue"

See User Guide for complete configuration options.

📚 Documentation

• Quick Start - Get started in 30 seconds
• User Guide - Complete user manual
• Installation Guide - Detailed installation instructions
• Performance Guide - Optimization details
• Benchmarking Guide - How to benchmark LightningGrep

🧪 Benchmarking

cd benchmarks
./run-all-benchmarks.sh

See Benchmarks README for details.

🏗️ Architecture

• Rust regex engine with SIMD optimizations
• NOT_STISLA integration - Quantum-inspired search algorithms
• Runtime CPU detection - Automatically selects best SIMD (AVX-512 > AVX2 > Scalar)
• Parallel execution - Lock-free work-stealing across CPU cores
• Memory optimization - Prefetching and huge pages support

🔬 Technical Details

Runtime CPU detection bypasses CPUID masking to detect AVX-512 support. Features cached per-boot for performance. Supports AVX2, AVX-512, AMX, and VNNI. Uses AVX-512 branchless search with opmask registers, software prefetching, and huge pages for optimal performance.

🎯 Use Cases

• Code search - Fast recursive search through codebases
• Log analysis - High-performance log file searching
• Large file search - Efficient search through multi-GB files
• Batch operations - Process multiple search queries efficiently

📝 License

Open Source License

LightningGrep is dual-licensed under:

• GNU Affero General Public License v3.0 - Copyleft open source license
• Commercial License - Available for proprietary use

Commercial Licensing

Commercial licensing is available for organizations that need:

• Proprietary use without AGPL obligations
• Commercial support and SLA guarantees
• Custom licensing terms
• Enterprise features and priority support

We're happy to discuss licensing regardless of your size or budget.

Our commercial licensing is designed to be fair and affordable:

• Startups & Small Businesses: Flexible terms with reasonable pricing
• Medical & Non-Profit Organizations: Special discounted rates
• Intelligence Community: Special consideration available for members of the intelligence community
• Enterprise: Volume discounts and custom SLA agreements

Contact us:

• Email: Commercial licensing inquiries
• GitHub Issues: Tag with [commercial-license] for public discussion
• Discussions: Open a discussion thread for questions

⚠️ License Compliance

LightningGrep includes license compliance features to ensure proper usage. License violations may result in legal action.

Home and personal use is always free and encouraged. Buy us a beer or shout us out if you find the tool useful! 🍺

🤝 Contributing

Contributions welcome! Please see our contributing guidelines and code of conduct.

🔗 Related Projects

• ripgrep - Original ripgrep (LightningGrep is based on)
• NOT_STISLA - Quantum-inspired search algorithms

📞 Support

• Issues: GitHub Issues
• Documentation: docs/
• Performance Questions: See Performance Documentation

---

LightningGrep - Search at the speed of light ⚡