Add self-contained error tracking system

[mensfeld]

Overview

Implements a comprehensive error tracking system for hookshot that captures application errors (not operational dispatch failures), with deduplication, aggregation, and an admin UI with resolution workflow.

Key Principle: This tracker focuses on unexpected application failures in controllers, non-dispatch jobs, and services. Dispatch errors remain operational cases handled by existing Delivery model logic.

Features

Core Components

1. Error Capture Service (Admin::Errors::Capture)

   • Generates fingerprint from error class + normalized message
   • Deduplicates by fingerprint with occurrence counting
   • Cleans backtraces (removes gem paths, limits to 50 lines)
   • Sanitizes context (removes passwords/tokens, 10KB limit)
   • Defensive error handling prevents capture failures from breaking app

2. Background Job (Admin::ErrorCaptureJob)

   • Async error capture via Solid Queue (non-blocking)
   • Uses discard_on to prevent infinite retry loops
   • Survives Puma restarts and is monitored via Mission Control

3. Database Model (Admin::ErrorRecord)

   • Stores: error_class, message, backtrace, context (JSON), fingerprint
   • Tracks: occurrences_count, first_occurred_at, last_occurred_at, resolved_at
   • Scopes: unresolved, resolved, recent_first
   • Indexes on fingerprint (unique), resolved_at, last_occurred_at

4. Admin UI (Available at /errors)

   • Three-tab view: Unresolved / Resolved / All
   • List view with occurrence badges (red if >10, yellow otherwise)
   • Detail view with backtrace highlighting, context JSON, timeline
   • Actions: Resolve, Unresolve, Delete, Delete All Resolved

5. Application Integration

   • Controllers: rescue_from in ApplicationController (production only)
   • Jobs: around_perform in ApplicationJob (excludes DispatchJob)
   • Exclusions: DispatchJob errors are operational, tracked via Delivery model

Maintenance

• Rake task: rake errors:cleanup deletes resolved errors older than 30 days
• Suggested cron: 0 2 * * * cd /app && bundle exec rake errors:cleanup

Implementation Details

Fingerprint Normalization

• Numbers → N
• UUIDs → UUID
• Hex addresses → 0xHEX
• Temp paths → /tmp/PATH

Safety Measures

1. Defensive rescue blocks in capture service
2. Async capture via Solid Queue (non-blocking)
3. Job discard_on prevents retry loops
4. Context sanitization removes sensitive keys
5. Controller capture only in production mode

Database Migration

Creates error_records table with:

• Fingerprint (unique index for deduplication)
• Occurrence tracking (count + timestamps)
• Resolution workflow (resolved_at timestamp)
• Indexes for query performance

Testing

Verified functionality:

• ✅ Error capture and storage
• ✅ Deduplication by fingerprint
• ✅ Occurrence count increments
• ✅ Async job processing
• ✅ Rake cleanup task
• ✅ Routes and UI paths

Routes

• GET /errors - List errors with tab filtering
• GET /errors/:id - Error detail view
• POST /errors/:id/resolve - Mark as resolved
• POST /errors/:id/unresolve - Mark as unresolved
• DELETE /errors/:id - Delete single error
• DELETE /errors/destroy_all - Bulk delete resolved errors

Future Enhancements

• Email notifications on critical errors (occurrences > threshold)
• Statistics badge showing unresolved count in navbar
• Search/filtering by error_class or date range
• CSV export for reporting