Add Prometheus monitoring and metrics instrumentation

[coderloganli]

Summary

This PR adds comprehensive Prometheus monitoring to GoChat, instrumenting all services with metrics collection for QPS, latency, connections, and RPC calls. The implementation includes:

• HTTP metrics: Request count, duration, in-flight requests by service/method/path
• RPC metrics: Server/client request tracking, duration histograms, in-flight requests
• Connection metrics: Active connections and total connection counts by type (WebSocket/TCP)
• Message metrics: Message send/receive counters
• User operation metrics: Login/register operation tracking
• Redis metrics: Operation counts by command type

Key Changes

New Packages

• pkg/metrics/metrics.go - Centralized Prometheus metric definitions (147 lines)
• pkg/metrics/server.go - Standalone metrics HTTP server for RPC services (42 lines)
• pkg/middleware/prometheus.go - Gin middleware for HTTP request instrumentation (35 lines)
• pkg/middleware/rpcx.go - RPCX plugin for RPC server instrumentation using goroutine ID tracking (68 lines)

Service Instrumentation

• API Service (api/) - Added Prometheus middleware and /metrics endpoint, instrumented user operations
• Logic Service (logic/) - Added metrics server on port 9091, instrumented RPC server and Redis operations
• Connect Services (connect/) - Metrics servers on ports 9092 (WebSocket) and 9093 (TCP), connection and message tracking
• Task Service (task/) - Metrics server on port 9094, RPC client metrics for calls to Connect services
• Site Service (site/) - Added /metrics endpoint on port 8080

Infrastructure

• deployments/prometheus/prometheus.yml - Prometheus scrape configuration for all services (51 lines)
• docker-compose.yml - Added Prometheus container, exposed metrics ports (9091-9094, 8080)
• Updated all services to expose metrics endpoints

Dependency Management

• Added github.com/prometheus/client_golang v1.14.0
• Removed vendor directory (3,470 files) - now using Go modules exclusively
• Updated .gitignore to exclude vendor/

Merged Changes from Master

• Upgraded to Go 1.23 (from Go 1.18)
• Updated Dockerfile to use golang:1.23 and debian:bookworm-slim
• Updated CI/CD workflow to Go 1.23 with branch name sanitization
• Fixed deprecated ioutil.ReadFile → os.ReadFile
• Updated netcat package to netcat-openbsd

Metrics Exposed

HTTP Metrics

• gochat_http_requests_total - Counter by service, method, path, status
• gochat_http_request_duration_seconds - Histogram by service, method, path
• gochat_http_requests_in_flight - Gauge by service

RPC Metrics

• gochat_rpc_server_requests_total - Counter by service, method, status
• gochat_rpc_server_duration_seconds - Histogram by service, method
• gochat_rpc_server_requests_in_flight - Gauge by service
• gochat_rpc_client_requests_total - Counter by service, method, status
• gochat_rpc_client_duration_seconds - Histogram by service, target, method

Connection Metrics

• gochat_connections_active - Gauge by service, type (websocket/tcp)
• gochat_connections_total - Counter by service, type, status

Message Metrics

• gochat_messages_total - Counter by service, direction (sent/received)

Application Metrics

• gochat_user_operations_total - Counter by operation (login/register), status
• gochat_redis_operations_total - Counter by service, command

Metrics Endpoints

Each service exposes metrics on the following endpoints:

• API: http://localhost:7070/metrics
• Logic: http://localhost:9091/metrics
• Connect WebSocket: http://localhost:9092/metrics
• Connect TCP: http://localhost:9093/metrics
• Task: http://localhost:9094/metrics
• Site: http://localhost:8080/metrics
• Prometheus UI: http://localhost:9090

Testing

Start Services

docker compose up -d

Verify Metrics

# Check individual service metrics
curl http://localhost:7070/metrics | grep gochat
curl http://localhost:9091/metrics | grep gochat
curl http://localhost:9092/metrics | grep gochat

# Access Prometheus UI
open http://localhost:9090

Example Queries

In Prometheus UI, try these queries:

# HTTP request rate by service
rate(gochat_http_requests_total[5m])

# P99 HTTP latency
histogram_quantile(0.99, rate(gochat_http_request_duration_seconds_bucket[5m]))

# Active connections
gochat_connections_active

# RPC call success rate
sum(rate(gochat_rpc_server_requests_total{status="success"}[5m])) by (service)

Test User Registration

# Register a user and check metrics
curl -X POST http://localhost:7070/user/register \
  -H "Content-Type: application/json" \
  -d '{"userName": "test", "password": "test123"}'

# Check user operation metrics
curl http://localhost:7070/metrics | grep gochat_user_operations_total

Bug Fixes

During implementation, fixed a critical RPCX plugin bug where PreCall was returning modified context instead of original args, causing reflection errors:

• Error: reflect: Call using *context.valueCtx as type *proto.RegisterRequest
• Fix: Changed to use sync.Map with goroutine ID for timing storage, returning args unchanged (see pkg/middleware/rpcx.go:40-47)

Files Changed

Modified: 23 files (+568, -54)

• Infrastructure: .github/workflows/ci-cd.yml, Makefile, docker-compose.yml, docker/Dockerfile
• Dependencies: go.mod, go.sum, .gitignore
• Services: api/, connect/, logic/, task/, site/
• New packages: pkg/metrics/, pkg/middleware/
• Configuration: deployments/prometheus/prometheus.yml

🤖 Generated with Claude Code