Skip to content

fix: Loki integration in release mode - logs now properly written and… #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bashizip merged 1 commit into from
Jan 30, 2026
Merged

fix: Loki integration in release mode - logs now properly written and… #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions docker/promtail-config-docker.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:
- url: http://loki:3100/loki/api/v1/push

scrape_configs:
# Scrape Docker container logs
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
# Only scrape containers in the abada-network
- source_labels: ['__meta_docker_network_name']
regex: 'abada-engine_abada-network'
action: keep

# Add container name as a label
- source_labels: ['__meta_docker_container_name']
target_label: 'container_name'

# Add service name from compose
- source_labels: ['__meta_docker_container_label_com_docker_compose_service']
target_label: 'service_name'

# Add project name
- source_labels: ['__meta_docker_container_label_com_docker_compose_project']
target_label: 'project'

# Set the log path to the container's log file
- source_labels: ['__meta_docker_container_id']
target_label: '__path__'
replacement: '/var/lib/docker/containers/$1/*-json.log'

# Keep the file-based scraping as fallback
- job_name: abada-engine-files
static_configs:
- targets:
- localhost
labels:
job: abada-engine-files
service_name: abada-engine
__path__: /var/log/abada/*.log
182 changes: 182 additions & 0 deletions docs/operations/loki-troubleshooting-resolution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
# Loki Integration Troubleshooting & Resolution

## Issue Summary
When running in release mode with Docker, logs were not appearing in the Grafana dashboard through Loki, and the label selector was empty.

## Root Causes Identified

### 1. **Logback Configuration Issue** (Primary)
The `logback-spring.xml` configuration had a critical bug where specific loggers (lines 56-78) were configured with:
- `additivity="false"`
- Only reference to `CONSOLE` appender
- **No reference to the `ASYNC_FILE` appender**

This meant that even though the prod profile's root logger was configured to use `ASYNC_FILE`, the specific loggers overrode this configuration and only wrote to console/stdout, not to file.

**Impact**: No logs were being written to `/app/logs/abada-engine.log`, so Promtail had nothing to scrape.

### 2. **Missing Volume Mount in Release Compose**
The `release/docker-compose.release.yml` didn't have a volume mount for the logs directory on the `abada-engine` service.

**Impact**: Even if logs were being written inside the container, they wouldn't be accessible to Promtail on the host filesystem.

## Solutions Implemented

### Solution 1: Fixed Logback Configuration
**File**: `src/main/resources/logback-spring.xml`

**Changes**:
- Split logger configurations into two `<springProfile>` blocks:
- `<springProfile name="!prod">`: Dev/test loggers - write only to CONSOLE
- `<springProfile name="prod">`: Production loggers - write to both JSON (console) and ASYNC_FILE

**Result**: In production mode, all logs are now written to both:
1. Console (as JSON for Docker logs)
2. File (`/app/logs/abada-engine.log`) for Promtail to scrape

### Solution 2: Added Volume Mount
**File**: `release/docker-compose.release.yml`

**Changes**:
Added volume mount to the `abada-engine` service:
```yaml
volumes:
- type: bind
source: /Users/pbash/repo/abada-platform/abada-engine/logs
target: /app/logs
bind: {}
```

**Result**: Logs written inside the container at `/app/logs/` are now accessible on the host at the `logs/` directory.

## Verification

### Check Logs Are Being Written
```bash
# Check log file size
wc -l logs/abada-engine.log

# View recent logs
tail -20 logs/abada-engine.log
```

### Check Promtail is Scraping
```bash
# View Promtail logs
docker logs promtail --tail 30

# Look for:
# - "tail routine: started"
# - "Seeked /var/log/abada/abada-engine.log"
```

### Query Loki API Directly
```bash
# Check available labels
curl -s 'http://localhost:3100/loki/api/v1/label' | jq .

# Expected output:
# {
# "status": "success",
# "data": [
# "filename",
# "job",
# "service_name"
# ]
# }

# Query logs
curl -s -G 'http://localhost:3100/loki/api/v1/query_range' \
--data-urlencode 'query={job="abada-engine"}' \
--data-urlencode 'limit=10' | jq .
```

### Check in Grafana
1. Open Grafana: http://localhost:3000
2. Go to **Explore** (compass icon in left sidebar)
3. Select **Loki** as data source
4. In the label filters, you should now see:
- `filename`
- `job`
- `service_name`
5. Use LogQL query: `{service_name="abada-engine"}`

## Alternative Solution: Docker Logging Driver (Optional)

A more cloud-native approach is to configure Promtail to scrape logs directly from Docker containers instead of files.

**File Created**: `docker/promtail-config-docker.yaml`

This configuration:
- Uses Docker service discovery
- Scrapes logs directly from `/var/lib/docker/containers/`
- Automatically adds labels from container metadata
- Requires mounting the Docker socket in Promtail

**To use this approach**:
1. Update `release/docker-compose.release.yml`:
```yaml
promtail:
volumes:
- type: bind
source: /var/run/docker.sock
target: /var/run/docker.sock
- type: bind
source: /Users/pbash/repo/abada-platform/abada-engine/docker/promtail-config-docker.yaml
target: /etc/promtail/config.yml
```

2. Restart Promtail:
```bash
docker compose -f release/docker-compose.release.yml restart promtail
```

## Current Architecture

```
┌─────────────────┐
│ Spring Boot │
│ Application │ (SPRING_PROFILES_ACTIVE=prod)
└────────┬────────┘
│
├─── Traces ────┐
├─── Metrics ───┤
├─── Logs (JSON to stdout)
└─── Logs (Plain text to file: /app/logs/abada-engine.log)
│
├─── Docker Logs (JSON)
└─── File System (Plain text)
│
▼
┌──────────┐
│ Promtail │ (scrapes file)
└────┬─────┘
│
┌──────────────────────┼────────────────────┐
│ │ │
▼ ▼ ▼
┌────────┐ ┌──────────┐ ┌──────┐
│ Jaeger │ │Prometheus│ │ Loki │
└────────┘ └──────────┘ └──────┘
│ │ │
└──────────────────────┴────────────────────┘
│
▼
┌──────────┐
│ Grafana │
└──────────┘
```

## Key Learnings

1. **Spring Boot Profile-specific Logging**: Logger configurations can be scoped to specific Spring profiles using `<springProfile>` tags
2. **Logger Additivity**: When `additivity="false"`, child loggers don't inherit appenders from parent loggers - you must explicitly configure all appenders
3. **File vs Docker Logging**: In containerized environments, consider whether to use file-based logging or Docker's native logging drivers
4. **Volume Mounts**: Always ensure log directories are mounted as volumes if you need external tools (like Promtail) to access them

## Related Files
- [logback-spring.xml](file:///Users/pbash/repo/abada-platform/abada-engine/src/main/resources/logback-spring.xml)
- [release/docker-compose.release.yml](file:///Users/pbash/repo/abada-platform/abada-engine/release/docker-compose.release.yml)
- [docker/promtail-config.yaml](file:///Users/pbash/repo/abada-platform/abada-engine/docker/promtail-config.yaml)
- [docker/promtail-config-docker.yaml](file:///Users/pbash/repo/abada-platform/abada-engine/docker/promtail-config-docker.yaml)
- [docker/loki-config-prod.yaml](file:///Users/pbash/repo/abada-platform/abada-engine/docker/loki-config-prod.yaml)
5 changes: 5 additions & 0 deletions release/docker-compose.release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ services:
abada-network: null
pull_policy: always
restart: always
volumes:
- type: bind
source: /Users/pbash/repo/abada-platform/abada-engine/logs
target: /app/logs
bind: {}
abada-orun:
container_name: abada-orun-prod
environment:
Expand Down
72 changes: 54 additions & 18 deletions src/main/resources/logback-spring.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,29 +53,65 @@
</appender>

<!-- Logger configurations -->
<logger name="com.abada.engine" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
<!-- Development Profile Loggers -->
<springProfile name="!prod">
<logger name="com.abada.engine" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>

<logger name="io.opentelemetry" level="INFO" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="io.opentelemetry" level="INFO" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>

<logger name="io.micrometer" level="INFO" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="io.micrometer" level="INFO" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>

<logger name="org.springframework.web" level="INFO" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="org.springframework.web" level="INFO" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>

<logger name="org.hibernate.SQL" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="org.hibernate.SQL" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>

<logger name="org.hibernate.type.descriptor.sql" level="TRACE" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="org.hibernate.type.descriptor.sql" level="TRACE" additivity="false">
<appender-ref ref="CONSOLE"/>
</logger>
</springProfile>

<!-- Production Profile Loggers -->
<springProfile name="prod">
<logger name="com.abada.engine" level="INFO" additivity="false">
<appender-ref ref="JSON"/>
<appender-ref ref="ASYNC_FILE"/>
</logger>

<logger name="io.opentelemetry" level="INFO" additivity="false">
<appender-ref ref="JSON"/>
<appender-ref ref="ASYNC_FILE"/>
</logger>

<logger name="io.micrometer" level="INFO" additivity="false">
<appender-ref ref="JSON"/>
<appender-ref ref="ASYNC_FILE"/>
</logger>

<logger name="org.springframework.web" level="INFO" additivity="false">
<appender-ref ref="JSON"/>
<appender-ref ref="ASYNC_FILE"/>
</logger>

<logger name="org.hibernate.SQL" level="INFO" additivity="false">
<appender-ref ref="JSON"/>
<appender-ref ref="ASYNC_FILE"/>
</logger>

<logger name="org.hibernate.type.descriptor.sql" level="INFO" additivity="false">
<appender-ref ref="JSON"/>
<appender-ref ref="ASYNC_FILE"/>
</logger>
</springProfile>

<!-- Root logger -->
<root level="INFO">
Expand Down
Loading