A secure, high-performance time-series database for LoRaWAN device data
LoRaDB is a specialized database built from scratch in Rust for storing and querying LoRaWAN network traffic. It features an LSM-tree storage engine, flexible ingestion (MQTT or HTTP), end-to-end encryption, and a simple query DSL.
- LSM-Tree Architecture: Write-Ahead Log (WAL) → Memtable → SSTables → Compaction
- Crash Recovery: CRC32-checksummed WAL entries with automatic replay
- Lock-Free Concurrency:
crossbeam-skiplistmemtable,DashMapdevice registry - Device-First Indexing: Composite key (DevEUI, timestamp, sequence) for efficient queries
- Bloom Filters: Probabilistic membership testing (1% false positive rate)
- LZ4 Compression: Efficient SSTable storage
- AES-256-GCM Encryption: Optional data-at-rest encryption with key zeroization
- Flexible Retention Policies: Global default + per-application retention with automatic enforcement
- Dual Network Support: ChirpStack v4 and The Things Network v3
- TLS 1.2+: Secure connections with system certificates
- Automatic Reconnection: Resilient connection handling
- Message Parsing: JSON deserialization with validation
- ChirpStack Webhook Support: Ingest data via HTTP webhooks when MQTT access is unavailable
- Can be used instead of or alongside MQTT ingestion
- Supported Events: Uplink, Join, and Status events
- Authenticated: JWT or API token required for all requests
- Same Data Model: HTTP-ingested data is queryable using the same DSL as MQTT data
- Use Cases: Helium networks, managed ChirpStack instances, webhook-based integrations
- See: HTTP Ingestion Guide for detailed setup instructions
Simple SQL-like query language with nested field projection:
-- Query all uplink data
SELECT * FROM device '0123456789ABCDEF' WHERE LAST '1h'
-- Query specific frame types
SELECT uplink FROM device '0123456789ABCDEF' WHERE SINCE '2025-01-01T00:00:00Z'
-- Query specific measurements using dot notation
SELECT decoded_payload.object.co2, decoded_payload.object.TempC_SHT FROM device '0123456789ABCDEF' WHERE LAST '24h'
-- Mix frame metadata and sensor measurements
SELECT received_at, f_port, decoded_payload.object.temperature FROM device '0123456789ABCDEF' WHERE LAST '7d'- Dual Authentication: JWT tokens (short-lived) + API tokens (long-lived, revocable)
- CORS Support: Configurable cross-origin resource sharing for web dashboards
- Security Headers: HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy
- TLS Support: Optional built-in TLS (use reverse proxy recommended for production)
- RESTful Endpoints:
GET /health- Health check (no auth)POST /ingest?event={type}- ChirpStack webhook ingestion (auth required)POST /query- Execute queries (auth required)GET /devices- List devices (auth required)GET /devices/:dev_eui- Device info (auth required)POST /tokens- Create API token (auth required)GET /tokens- List API tokens (auth required)DELETE /tokens/:token_id- Revoke API token (auth required)GET /retention/policies- List retention policies (auth required)POST /retention/enforce- Trigger retention enforcement (auth required)
LoRaDB includes automated deployment scripts for easy setup and updates:
# Clone repository
git clone https://github.com/yourusername/loradb
cd loradb
# Deploy (handles everything)
./deploy.shThe deploy.sh script will:
- Validate configuration
- Build Docker image
- Create volumes
- Start LoRaDB
- Show next steps
When code updates are available:
# Pull changes and rebuild
./update.shThe update.sh script will:
- Pull latest changes from git
- Show what's new
- Rebuild Docker image
- Restart with data persistence
- Verify health
Use the helper script for common operations:
# View all available commands
./loradb.sh
# Common commands
./loradb.sh logs # Follow logs
./loradb.sh status # Check status
./loradb.sh token admin # Generate JWT token
./loradb.sh apitoken admin "My Dashboard" 365 # Generate API token
./loradb.sh backup # Create backup
./loradb.sh health # Check API healthSee DEPLOYMENT.md for complete deployment guide.
Recommended for low-powered devices (Raspberry Pi, etc.)
Pre-built Docker images are automatically published to GitHub Container Registry. This completely avoids building on your device.
- Clone the repository
git clone https://github.com/yourusername/loradb
cd loradb- Create environment configuration
cp .env.example .env
# Edit .env with your configuration- Use the pre-built image
# Option A: Use the pre-built compose file
docker-compose -f docker-compose.prebuilt.yml up -d
# Option B: Pull and run manually
docker pull ghcr.io/yourusername/loradb:latest
docker run -d --name loradb \
--env-file .env \
-p 8443:8443 \
-v loradb-data:/var/lib/loradb/data \
ghcr.io/yourusername/loradb:latest- Update to latest version
docker-compose -f docker-compose.prebuilt.yml pull
docker-compose -f docker-compose.prebuilt.yml up -dlatest- Latest commit on main branchv0.1.0- Specific version tags0.1- Major.minor versionsha-abc123- Specific commit SHA
Pre-built images support both:
linux/amd64- x86_64 systems (standard servers/desktops)linux/arm64- ARM64 systems (Raspberry Pi 3/4/5, AWS Graviton)
Docker automatically pulls the correct architecture for your device.
- Docker 20.10+
- Docker Compose 2.0+
- (Optional) Reverse proxy like Caddy or nginx for production HTTPS
- Clone the repository
git clone https://github.com/yourusername/loradb
cd loradb- Create environment configuration
cp .env.example .env- Edit
.envwith your configuration
# Required: Generate a secure JWT secret
LORADB_API_JWT_SECRET=$(openssl rand -base64 32)
# Optional: Configure MQTT broker (ChirpStack or TTN) - or use HTTP ingestion instead
LORADB_MQTT_CHIRPSTACK_BROKER=mqtts://chirpstack.example.com:8883
LORADB_MQTT_USERNAME=loradb
LORADB_MQTT_PASSWORD=your-password
# Optional: Use reverse proxy for HTTPS (recommended)
LORADB_API_BIND_ADDR=0.0.0.0:8080
LORADB_API_ENABLE_TLS=false- Start the container
docker-compose up -d- View logs
docker-compose logs -f loradb- Stop the container
docker-compose down- Minimum: 512MB RAM, 1 CPU core, 10GB disk
- Recommended: 2GB RAM, 2 CPU cores, 50GB+ SSD
For production deployments, use a reverse proxy like Caddy or nginx to handle HTTPS:
With Caddy (automatic HTTPS):
# Update .env
LORADB_API_BIND_ADDR=0.0.0.0:8080
LORADB_API_ENABLE_TLS=false
# Caddy will automatically obtain Let's Encrypt certificates
# and proxy to LoRaDB on port 8080Benefits:
- ✅ Automatic HTTPS with Let's Encrypt
- ✅ Certificate renewal handled by Caddy
- ✅ Easier configuration
- ✅ Better performance for static assets
- ✅ Additional security features (rate limiting, etc.)
LoRaDB uses an LSM-tree storage engine with multiple persistence layers:
- Write-Ahead Log (WAL): All writes are immediately logged to
wal/directory for crash recovery - Memtable: In-memory sorted data structure (flushed periodically or when size threshold is reached)
- SSTables: Immutable sorted files (
sstable-*.sst) created when memtable is flushed
When SSTables are created:
- Every 5 minutes (configurable via
LORADB_STORAGE_MEMTABLE_FLUSH_INTERVAL_SECS) - When memtable reaches 64MB (configurable via
LORADB_STORAGE_MEMTABLE_SIZE_MB) - On graceful shutdown (SIGTERM/SIGINT)
Data directory structure:
/var/lib/loradb/data/
├── wal/ # Write-ahead logs
│ └── segment-*.wal
├── sstable-*.sst # Sorted string tables (persistent data)
└── api_tokens.json # API token store
Data is persisted in the loradb-data Docker volume.
Backup procedure:
# Create backup (container can be running)
docker run --rm -v loradb_loradb-data:/data -v $(pwd):/backup \
alpine tar czf /backup/loradb-backup.tar.gz -C /data .Restore procedure:
# IMPORTANT: Stop the container first
docker compose down
# Clear existing data (optional but recommended)
docker run --rm -v loradb_loradb-data:/data alpine rm -rf /data/*
# Restore backup
docker run --rm -v loradb_loradb-data:/data -v $(pwd):/backup \
alpine tar xzf /backup/loradb-backup.tar.gz -C /data
# Start the container
docker compose up -dWhy stop the container? If the container is running during restore, it will immediately compact the restored SSTables into new files and delete the originals, resulting in data loss.
- Rust 1.70+ (2021 edition)
- OpenSSL development libraries
- Optional: TLS certificates for HTTPS
git clone https://github.com/yourusername/loradb
cd loradb
cargo build --releasetarget/release/loradb
LoRaDB is configured via environment variables or a .env file:
# Storage
LORADB_STORAGE_DATA_DIR=/var/lib/loradb/data
# API
LORADB_API_BIND_ADDR=0.0.0.0:8080
LORADB_API_JWT_SECRET=your-32-character-secret-here!!!
# TLS Configuration (optional - use reverse proxy like Caddy/nginx for production)
LORADB_API_ENABLE_TLS=false # Set to true for direct HTTPS
# LORADB_API_TLS_CERT=/path/to/cert.pem # Only needed if ENABLE_TLS=true
# LORADB_API_TLS_KEY=/path/to/key.pem # Only needed if ENABLE_TLS=true# MQTT - ChirpStack
LORADB_MQTT_CHIRPSTACK_BROKER=mqtts://chirpstack.example.com:8883
LORADB_MQTT_USERNAME=loradb
LORADB_MQTT_PASSWORD=secret
# MQTT - The Things Network
LORADB_MQTT_TTN_BROKER=mqtts://nam1.cloud.thethings.network:8883
# Storage Tuning
LORADB_STORAGE_WAL_SYNC_INTERVAL_MS=1000
LORADB_STORAGE_MEMTABLE_SIZE_MB=64
LORADB_STORAGE_MEMTABLE_FLUSH_INTERVAL_SECS=300 # Periodic flush every 5 minutes
LORADB_STORAGE_COMPACTION_THRESHOLD=10
# Data Retention Policies (optional - defaults to keep forever)
LORADB_STORAGE_RETENTION_DAYS=90 # Global default: delete data older than 90 days
LORADB_STORAGE_RETENTION_APPS="test-app:7,production:365,critical:never" # Per-application policies
LORADB_STORAGE_RETENTION_CHECK_INTERVAL_HOURS=24 # How often to enforce retention
# Encryption (optional)
LORADB_STORAGE_ENABLE_ENCRYPTION=true
LORADB_STORAGE_ENCRYPTION_KEY=base64-encoded-32-byte-key
# API Tuning
LORADB_API_JWT_EXPIRATION_HOURS=1 # JWT token expiration in hours (default: 1)
LORADB_API_RATE_LIMIT_PER_MINUTE=100
LORADB_API_CORS_ALLOWED_ORIGINS=* # CORS allowed origins (* for dev, specific domains for prod)# Using environment variables
export LORADB_STORAGE_DATA_DIR=/var/lib/loradb/data
export LORADB_API_BIND_ADDR=0.0.0.0:8443
export LORADB_API_JWT_SECRET=your-secret-key-at-least-32-chars
# ... other variables ...
./target/release/loradbLoRaDB includes a built-in token generator tool for easy authentication.
# Generate a token for a user
docker compose exec loradb generate-token admin
# Output example:
# Generated JWT token for user 'admin':
#
# eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9...
#
# Use this token in API requests:
# curl -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9...' https://your-domain.com/devices# Build the token generator
cargo build --release --bin generate-token
# Generate token using JWT secret from environment
export LORADB_API_JWT_SECRET="your-32-character-secret-key-here"
./target/release/generate-token admin
# Or pass JWT secret directly
./target/release/generate-token admin "your-32-character-secret-key-here"
# Generate token with custom expiration (in hours)
export LORADB_API_JWT_EXPIRATION_HOURS=24 # 24 hours
./target/release/generate-token admin
# Or pass expiration as third argument
./target/release/generate-token admin "your-jwt-secret" 24- Algorithm: HS256 (HMAC with SHA-256)
- Expiration: Configurable via
LORADB_API_JWT_EXPIRATION_HOURS(default: 1 hour) - Claims: Contains
sub(username),exp(expiration), andiat(issued at) - Usage: Include in API requests via
Authorization: Bearer <token>header
# Health check
curl https://localhost:8443/health
# Execute query
curl -X POST https://localhost:8443/query \
-H "Authorization: Bearer YOUR_JWT_TOKEN" \
-H "Content-Type: application/json" \
-d '{"query": "SELECT * FROM device '\''0123456789ABCDEF'\'' WHERE LAST '\''1h'\''"}'
# List devices
curl https://localhost:8443/devices \
-H "Authorization: Bearer YOUR_JWT_TOKEN"
# Get device info
curl https://localhost:8443/devices/0123456789ABCDEF \
-H "Authorization: Bearer YOUR_JWT_TOKEN"┌─────────────────────────────────────────────────────┐
│ MQTT Brokers │
│ (ChirpStack v4, TTN v3) │
└──────────────────┬──────────────────────────────────┘
│ TLS 1.2+
▼
┌─────────────────┐
│ MQTT Ingestor │
│ - TLS Connect │
│ - Parse JSON │
└────────┬────────┘
│ mpsc channel
▼
┌─────────────────────────┐
│ Storage Engine │
│ ┌──────────────────┐ │
│ │ WAL (CRC32) │ │
│ └──────────────────┘ │
│ ┌──────────────────┐ │
│ │ Memtable │ │
│ │ (skiplist) │ │
│ └──────────────────┘ │
│ ┌──────────────────┐ │
│ │ SSTables │ │
│ │ (LZ4 + Bloom) │ │
│ └──────────────────┘ │
│ ┌──────────────────┐ │
│ │ Compaction │ │
│ └──────────────────┘ │
└─────────┬───────────────┘
│
▼
┌─────────────────┐
│ Query Executor │
│ - Parse DSL │
│ - Filter │
│ - Project │
└────────┬────────┘
│
▼
┌─────────────────────────┐
│ HTTPS API Server │
│ - JWT Auth Middleware │
│ - Security Headers │
│ - TLS (rustls) │
└─────────────────────────┘
│
▼
┌──────────┐
│ Client │
└──────────┘
- ✅ TLS 1.2+ for MQTT and HTTPS
- ✅ Dual Authentication (JWT + API tokens with revocation)
- ✅ Configurable CORS with origin restrictions
- ✅ Security Headers: HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy
- ✅ AES-256-GCM encryption-at-rest (optional)
- ✅ Key Zeroization on drop
- ✅ No
unsafecode (except dependencies) - ✅ Strict file permissions (0600/0700)
- Generate strong JWT secrets:
openssl rand -base64 32 - Use proper TLS certificates: Let's Encrypt or internal CA
- Enable encryption: Set
LORADB_STORAGE_ENABLE_ENCRYPTION=true - Restrict CORS origins:
# Development (allow all) LORADB_API_CORS_ALLOWED_ORIGINS=* # Production (specific origins only) LORADB_API_CORS_ALLOWED_ORIGINS=https://dashboard.example.com,https://admin.example.com
- Use API tokens for dashboards: Long-lived, revocable tokens for automation
- Monitor logs: Use structured JSON logging
- Rate limiting: Configure per deployment needs
# Run all tests
cargo test
# Run specific test suite
cargo test --lib storage
cargo test --lib api
cargo test --lib query
# With output
cargo test -- --nocapture
# Test coverage summary
cargo test 2>&1 | grep "test result"Test Results: 75 tests passing ✅
- Storage Engine: 19 tests (WAL, Memtable, SSTable, Compaction)
- Security: 18 tests (Encryption 7, JWT 11)
- Query System: 16 tests (DSL 4, Parser 8, Executor 4)
- API Layer: 11 tests (Handlers 3, Middleware 4, HTTP 4)
- MQTT: 2 tests
- Device Registry: 1 test
- Model: 8 tests
# Larger memtable for high ingestion rates
LORADB_STORAGE_MEMTABLE_SIZE_MB=128
# Less frequent WAL syncs (higher throughput, lower durability)
LORADB_STORAGE_WAL_SYNC_INTERVAL_MS=5000# Trigger compaction with more SSTables (less frequent compaction)
LORADB_STORAGE_COMPACTION_THRESHOLD=20- Write Throughput: ~10,000 frames/sec (unencrypted), ~5,000 frames/sec (encrypted)
- Query Latency: <100ms for 1M frames, device-scoped
- Storage Efficiency: ~60% compression ratio with LZ4
LoRaDB supports flexible retention policies to automatically delete old data based on configured retention periods. This enables compliance with data retention requirements, cost optimization, and privacy regulations.
Set a default retention period for all applications:
# Delete all data older than 90 days
LORADB_STORAGE_RETENTION_DAYS=90
# Check and enforce retention policy daily
LORADB_STORAGE_RETENTION_CHECK_INTERVAL_HOURS=24Override the global default with application-specific policies:
# Global default: 90 days
LORADB_STORAGE_RETENTION_DAYS=90
# Per-application overrides
LORADB_STORAGE_RETENTION_APPS="test-sensors:7,production:365,fire-alarms:never"Development vs Production:
LORADB_STORAGE_RETENTION_APPS="dev:7,staging:14,test:7,production:365"Privacy Compliance (GDPR/HIPAA):
# Occupancy data (30 days for privacy)
# HVAC data (1 year for energy analysis)
# Fire alarms (forever for compliance)
LORADB_STORAGE_RETENTION_APPS="occupancy:30,hvac:365,fire-alarms:never,smoke-alarms:never"Multi-Tenant SaaS:
# Different retention tiers for different customers
LORADB_STORAGE_RETENTION_APPS="customer-basic:30,customer-premium:365,customer-enterprise:730"Cost Optimization:
# Quick cleanup for test data, longer retention for production analytics
LORADB_STORAGE_RETENTION_APPS="test:3,staging:7,prod-monitoring:90,prod-analytics:365"- Application Policy Lookup: For each SSTable, retrieves all application IDs it contains
- Policy Resolution: Checks per-application policy → falls back to global default
- Conservative Deletion: Uses the longest retention period among all apps in the SSTable
- Never Override: If any application is set to
never, the entire SSTable is preserved - Automatic Enforcement: Background task runs at configured interval (default: 24 hours)
application-id:days Delete after specified days
application-id:never Keep forever (never delete)
Multiple policies are comma-separated:
LORADB_STORAGE_RETENTION_APPS="app1:30,app2:90,app3:never,app4:7"NEW: Retention policies can now be managed dynamically via REST API without server restart!
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8080/retention/policiesResponse:
{
"global_days": 90,
"check_interval_hours": 24,
"applications": [
{
"application_id": "production",
"days": 365,
"created_at": "2025-01-26T12:00:00Z",
"updated_at": "2025-01-26T12:00:00Z"
}
]
}# Get global policy
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8080/retention/policies/global
# Set global policy to 90 days
curl -X PUT -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"days": 90}' \
http://localhost:8080/retention/policies/global
# Set to "never" (keep forever)
curl -X PUT -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"days": null}' \
http://localhost:8080/retention/policies/global# Set retention for specific application
curl -X PUT -H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"days": 30}' \
http://localhost:8080/retention/policies/test-sensors
# Get application-specific policy
curl -H "Authorization: Bearer $TOKEN" \
http://localhost:8080/retention/policies/test-sensors
# Remove application policy (falls back to global)
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
http://localhost:8080/retention/policies/test-sensors# Run retention enforcement immediately (instead of waiting for scheduled run)
curl -X POST -H "Authorization: Bearer $TOKEN" \
http://localhost:8080/retention/enforceBenefits:
- No Restart Required: Update policies on the fly
- Auditable: Track when policies were created/updated
- Integration-Friendly: Automate retention management via API
- Backward Compatible: Environment variables still work; API takes precedence
Storage Location: Policies are persisted in <data_dir>/retention_policies.json
LoRaDB is designed for edge compatibility:
Docker deployment works seamlessly on edge devices:
# On Raspberry Pi 4 or similar ARM64 devices
docker-compose up -d
# Monitor resource usage
docker stats loradb
# Adjust resource limits in docker-compose.yml if neededEdge-specific Docker configuration:
- Reduce
LORADB_STORAGE_MEMTABLE_SIZE_MBto 32 for devices with limited RAM - Set appropriate CPU and memory limits in
docker-compose.yml - Use external USB/SSD storage for the data volume on Raspberry Pi
# Cross-compile for ARM64
rustup target add aarch64-unknown-linux-gnu
cargo build --release --target aarch64-unknown-linux-gnuBuilding ARM64 Docker image:
# On x86_64 host with buildx
docker buildx build --platform linux/arm64 -t loradb:arm64 .
# Or build natively on ARM64 device
docker build -t loradb .- Minimum: 512MB RAM, 1 CPU core, 10GB disk
- Recommended: 2GB RAM, 2 CPU cores, 50GB+ SSD
- ✅ x86_64 Linux (Docker & native)
- ✅ ARM64 Linux (Raspberry Pi 4, AWS Graviton) (Docker & native)
- ✅ Docker on edge gateways
⚠️ macOS (development only, not production)
Container won't start:
# Check logs
docker-compose logs loradb
# Verify environment variables
docker-compose config
# Check certificate mounts
docker exec loradb ls -l /etc/loradb/Permission errors:
# Ensure certificate files are readable
chmod 644 /path/to/cert.pem
chmod 600 /path/to/key.pem
# Check data volume permissions
docker exec loradb ls -ld /var/lib/loradb/dataHealth check failing:
# Test health endpoint manually
docker exec loradb curl -k https://localhost:8443/health
# Check if TLS certificates are valid
openssl x509 -in /path/to/cert.pem -text -noout# From host
openssl s_client -connect chirpstack.example.com:8883
# From container
docker exec loradb sh -c "apk add openssl && openssl s_client -connect chirpstack.example.com:8883"
# Check MQTT credentials in .env
grep MQTT .env# Check data directory permissions (inside container)
docker exec loradb ls -ld /var/lib/loradb/data # Should be owned by loradb user
# Check WAL recovery
docker-compose logs loradb | grep "Recovered"
# Inspect data volume
docker volume inspect loradb_loradb-data
# Native deployment
ls -ld /var/lib/loradb/data # Should be 0700# Verify JWT secret length (must be ≥32 chars)
echo -n "$LORADB_API_JWT_SECRET" | wc -c
# Test token generation with correct algorithm (HS256)
# Docker: Test API access
curl -k https://localhost:8443/health- ❌ No WASM/JavaScript payload decoders (use pre-decoded from network server)
- ❌ No clustering/replication (single-node only)
- ❌ No time-series aggregation functions (use external tools)
- Multi-node clustering
- Aggregate functions (AVG, MIN, MAX, COUNT)
- Grafana datasource plugin
- Prometheus metrics exporter
MIT License - See LICENSE file for details
Contributions welcome! Please:
- Run
cargo testbefore submitting - Follow Rust idioms and style guidelines
- Add tests for new features
- Update documentation
- Issues: https://github.com/yourusername/loradb/issues
- Discussions: https://github.com/yourusername/loradb/discussions
- Security: security@yourdomain.com
Built with: