diff --git a/docker/promtail-config-docker.yaml b/docker/promtail-config-docker.yaml new file mode 100644 index 0000000..8febacd --- /dev/null +++ b/docker/promtail-config-docker.yaml @@ -0,0 +1,48 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +clients: + - url: http://loki:3100/loki/api/v1/push + +scrape_configs: + # Scrape Docker container logs + - job_name: docker + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 5s + relabel_configs: + # Only scrape containers in the abada-network + - source_labels: ['__meta_docker_network_name'] + regex: 'abada-engine_abada-network' + action: keep + + # Add container name as a label + - source_labels: ['__meta_docker_container_name'] + target_label: 'container_name' + + # Add service name from compose + - source_labels: ['__meta_docker_container_label_com_docker_compose_service'] + target_label: 'service_name' + + # Add project name + - source_labels: ['__meta_docker_container_label_com_docker_compose_project'] + target_label: 'project' + + # Set the log path to the container's log file + - source_labels: ['__meta_docker_container_id'] + target_label: '__path__' + replacement: '/var/lib/docker/containers/$1/*-json.log' + + # Keep the file-based scraping as fallback + - job_name: abada-engine-files + static_configs: + - targets: + - localhost + labels: + job: abada-engine-files + service_name: abada-engine + __path__: /var/log/abada/*.log diff --git a/docs/operations/loki-troubleshooting-resolution.md b/docs/operations/loki-troubleshooting-resolution.md new file mode 100644 index 0000000..b1081ec --- /dev/null +++ b/docs/operations/loki-troubleshooting-resolution.md @@ -0,0 +1,182 @@ +# Loki Integration Troubleshooting & Resolution + +## Issue Summary +When running in release mode with Docker, logs were not appearing in the Grafana dashboard through Loki, and the label selector was empty. + +## Root Causes Identified + +### 1. **Logback Configuration Issue** (Primary) +The `logback-spring.xml` configuration had a critical bug where specific loggers (lines 56-78) were configured with: +- `additivity="false"` +- Only reference to `CONSOLE` appender +- **No reference to the `ASYNC_FILE` appender** + +This meant that even though the prod profile's root logger was configured to use `ASYNC_FILE`, the specific loggers overrode this configuration and only wrote to console/stdout, not to file. + +**Impact**: No logs were being written to `/app/logs/abada-engine.log`, so Promtail had nothing to scrape. + +### 2. **Missing Volume Mount in Release Compose** +The `release/docker-compose.release.yml` didn't have a volume mount for the logs directory on the `abada-engine` service. + +**Impact**: Even if logs were being written inside the container, they wouldn't be accessible to Promtail on the host filesystem. + +## Solutions Implemented + +### Solution 1: Fixed Logback Configuration +**File**: `src/main/resources/logback-spring.xml` + +**Changes**: +- Split logger configurations into two `` blocks: + - ``: Dev/test loggers - write only to CONSOLE + - ``: Production loggers - write to both JSON (console) and ASYNC_FILE + +**Result**: In production mode, all logs are now written to both: +1. Console (as JSON for Docker logs) +2. File (`/app/logs/abada-engine.log`) for Promtail to scrape + +### Solution 2: Added Volume Mount +**File**: `release/docker-compose.release.yml` + +**Changes**: +Added volume mount to the `abada-engine` service: +```yaml +volumes: + - type: bind + source: /Users/pbash/repo/abada-platform/abada-engine/logs + target: /app/logs + bind: {} +``` + +**Result**: Logs written inside the container at `/app/logs/` are now accessible on the host at the `logs/` directory. + +## Verification + +### Check Logs Are Being Written +```bash +# Check log file size +wc -l logs/abada-engine.log + +# View recent logs +tail -20 logs/abada-engine.log +``` + +### Check Promtail is Scraping +```bash +# View Promtail logs +docker logs promtail --tail 30 + +# Look for: +# - "tail routine: started" +# - "Seeked /var/log/abada/abada-engine.log" +``` + +### Query Loki API Directly +```bash +# Check available labels +curl -s 'http://localhost:3100/loki/api/v1/label' | jq . + +# Expected output: +# { +# "status": "success", +# "data": [ +# "filename", +# "job", +# "service_name" +# ] +# } + +# Query logs +curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \ + --data-urlencode 'query={job="abada-engine"}' \ + --data-urlencode 'limit=10' | jq . +``` + +### Check in Grafana +1. Open Grafana: http://localhost:3000 +2. Go to **Explore** (compass icon in left sidebar) +3. Select **Loki** as data source +4. In the label filters, you should now see: + - `filename` + - `job` + - `service_name` +5. Use LogQL query: `{service_name="abada-engine"}` + +## Alternative Solution: Docker Logging Driver (Optional) + +A more cloud-native approach is to configure Promtail to scrape logs directly from Docker containers instead of files. + +**File Created**: `docker/promtail-config-docker.yaml` + +This configuration: +- Uses Docker service discovery +- Scrapes logs directly from `/var/lib/docker/containers/` +- Automatically adds labels from container metadata +- Requires mounting the Docker socket in Promtail + +**To use this approach**: +1. Update `release/docker-compose.release.yml`: + ```yaml + promtail: + volumes: + - type: bind + source: /var/run/docker.sock + target: /var/run/docker.sock + - type: bind + source: /Users/pbash/repo/abada-platform/abada-engine/docker/promtail-config-docker.yaml + target: /etc/promtail/config.yml + ``` + +2. Restart Promtail: + ```bash + docker compose -f release/docker-compose.release.yml restart promtail + ``` + +## Current Architecture + +``` +┌─────────────────┐ +│ Spring Boot │ +│ Application │ (SPRING_PROFILES_ACTIVE=prod) +└────────┬────────┘ + │ + ├─── Traces ────┐ + ├─── Metrics ───┤ + ├─── Logs (JSON to stdout) + └─── Logs (Plain text to file: /app/logs/abada-engine.log) + │ + ├─── Docker Logs (JSON) + └─── File System (Plain text) + │ + ▼ + ┌──────────┐ + │ Promtail │ (scrapes file) + └────┬─────┘ + │ + ┌──────────────────────┼────────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌────────┐ ┌──────────┐ ┌──────┐ + │ Jaeger │ │Prometheus│ │ Loki │ + └────────┘ └──────────┘ └──────┘ + │ │ │ + └──────────────────────┴────────────────────┘ + │ + ▼ + ┌──────────┐ + │ Grafana │ + └──────────┘ +``` + +## Key Learnings + +1. **Spring Boot Profile-specific Logging**: Logger configurations can be scoped to specific Spring profiles using `` tags +2. **Logger Additivity**: When `additivity="false"`, child loggers don't inherit appenders from parent loggers - you must explicitly configure all appenders +3. **File vs Docker Logging**: In containerized environments, consider whether to use file-based logging or Docker's native logging drivers +4. **Volume Mounts**: Always ensure log directories are mounted as volumes if you need external tools (like Promtail) to access them + +## Related Files +- [logback-spring.xml](file:///Users/pbash/repo/abada-platform/abada-engine/src/main/resources/logback-spring.xml) +- [release/docker-compose.release.yml](file:///Users/pbash/repo/abada-platform/abada-engine/release/docker-compose.release.yml) +- [docker/promtail-config.yaml](file:///Users/pbash/repo/abada-platform/abada-engine/docker/promtail-config.yaml) +- [docker/promtail-config-docker.yaml](file:///Users/pbash/repo/abada-platform/abada-engine/docker/promtail-config-docker.yaml) +- [docker/loki-config-prod.yaml](file:///Users/pbash/repo/abada-platform/abada-engine/docker/loki-config-prod.yaml) diff --git a/release/docker-compose.release.yml b/release/docker-compose.release.yml index b287ded..300ed0d 100644 --- a/release/docker-compose.release.yml +++ b/release/docker-compose.release.yml @@ -52,6 +52,11 @@ services: abada-network: null pull_policy: always restart: always + volumes: + - type: bind + source: /Users/pbash/repo/abada-platform/abada-engine/logs + target: /app/logs + bind: {} abada-orun: container_name: abada-orun-prod environment: diff --git a/src/main/resources/logback-spring.xml b/src/main/resources/logback-spring.xml index 7f04b5b..00c2648 100644 --- a/src/main/resources/logback-spring.xml +++ b/src/main/resources/logback-spring.xml @@ -53,29 +53,65 @@ - - - + + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +