Add Observability: Metrics for the Project

[sthanikan2000]

Problem

Currently, the project does not have any performance metrics or monitoring in place. This makes it difficult to understand how well the application is performing, identify bottlenecks, and track improvements over time.
To ensure long-term maintainability and observability, we need to define and implement key performance metrics.

Proposed Solution

• Identify key performance metrics relevant to the project (e.g., latency, response time, error rate, memory usage, CPU usage)
• Integrate monitoring tools (e.g., Prometheus, Grafana, OpenTelemetry, or simple logging-based metrics)
• Add instrumentation where needed (API endpoints, background jobs, database queries, etc.)
• Provide a dashboard or simple metric reports
• Document how to monitor and interpret the metrics

Completed

• Refactor Observability stack to be vendor agnostic 317 Refactor Observability Stack with OpenTelemetry #371
• Connect to PDP and OE 317 Add Observability to PDP and OE #408
• Connect to Audit Service

No response

[ginaxu1]

Implementation Plan

Phase 1 – OTel SDK & Base Metrics

1. Add Go module dependencies for go.opentelemetry.io/otel, otel/sdk/metric, and the Prometheus exporter.
2. Initialize an OTel SDK in each service: create a metrics provider, register the Prometheus exporter, and expose /metrics.
3. Add an HTTP middleware (or gRPC interceptor) that records latency, request counts, and error status using OTel instruments.
4. Enable OTel runtime instrumentation to record goroutine count, GC stats, heap usage, etc.

Phase 2 – Custom Instrumentation

5. Wrap all outbound calls (DB, HTTP clients, etc.) with OTel timers/counters to capture latency & failures.
6. Pick 2–3 business-critical flows (e.g., PDP job processing, schema publish) and add domain metrics such as pdp_jobs_processed_total, schema_publish_failures_total.

Phase 3 – Monitoring Infrastructure

7. Provision Prometheus (local or managed) and configure it to scrape the new /metrics endpoint(s).
8. Stand up a trace backend (Jaeger, Tempo, OTLP collector) and wire the Go services to export traces.
9. Create Grafana dashboards for the four golden signals (latency, traffic, errors, saturation) plus key business counters.
10. Define alerting rules (Prometheus / Alertmanager) for high P99 latency, elevated error rates, CPU saturation, etc., and document how to interpret/triage those alerts.

Documentation / Ops

11. Add README/ops docs covering how to run Prometheus/Grafana locally, where to view dashboards, and how to enable tracing exporters per environment.

[ginaxu1]

#319