Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1 @@
nomad_follower
nomad_logfetcher
29 changes: 29 additions & 0 deletions Containerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
FROM docker.io/library/golang:1.24 as base

WORKDIR /app

COPY ./go.mod ./go.sum ./logfetcher/
WORKDIR /app/logfetcher
RUN go mod download;

FROM base as dev
RUN go install github.com/air-verse/air@latest

# volume is mounted by compose for hot reload
COPY ./ ./

ENTRYPOINT [ \
"air", \
"--build.poll=true", \
"--build.send_interrupt=true", \
"--build.kill_delay=1", \
"--build.cmd", "/usr/bin/true", \
"--build.bin", "go run ./" \
]

### Build
FROM base as logfetcher
COPY ./ ./
RUN CGO_ENABLED=0 GOOS=linux go build -mod=mod -o /logfetcher ./

ENTRYPOINT [ "/logfetcher" ]
7 changes: 0 additions & 7 deletions Dockerfile

This file was deleted.

16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
# nomad_follower
# nomad_logfetcher
Log forwarder for aggregating allocation logs from nomad worker agents.

## Running the application
Run the application on each worker in a nomad cluster.
```
docker pull sofixa/nomad_follower:latest
docker run -v log_folder:/log -e LOG_TAG="logging" "-e LOG_FILE="/log/nomad-forwarder.log" sofixa/nomad_follower:latest
podman pull Aiven-Open/nomad_logfetcher:latest
podman run -v log_folder:/log -e LOG_TAG="logging" -e LOG_FILE="/log/nomad-forwarder.log" Aiven-Open/nomad_logfetcher:latest
```

Example Nomad job file available [here](./nomad_follower.nomad), with an example job outputting random logs [here](./example.nomad).
Example Nomad job file available [here](./nomad_logfetcher.nomad), with an example job outputting random logs [here](./example.nomad).

nomad_follower:
- will follow all allocations if it contains service tag ```nomad_follower``` (can be changed with `LOG_TAG` environment variable) on the worker and tail the allocation logs to the aggregate log file.
nomad_logfetcher:
- will follow all allocations if it contains service tag ```nomad_logfetcher``` (can be changed with `LOG_TAG` environment variable) on the worker and tail the allocation logs to the aggregate log file.
- will stop following completed allocations and will start following new allocations as they become available.
- can be deployed with nomad in a system task group along with a log collector. The aggregate log file can then be shared with the log collector by writing the aggregate log file into the shared allocation folder.
- formats log entries as json formatted logs. It will convert string formatted logs to json formatted logs by passing the log entry in the ```message``` key.
- adds a ```service_name``` key that contains the listed service names for a task.

Using nomad_follower prevents the cluster operator from having to run a log collector in every task group for every task on a worker while still allowing nomad to handle the logs for each allocation.
Using nomad_logfetcher prevents the cluster operator from having to run a log collector in every task group for every task on a worker while still allowing nomad to handle the logs for each allocation.

# License and attribution

This project is a fork of [this fork](https://github.com/sas1024/nomad_follower) of this original [project](https://github.com/adragoset/nomad_follower).
This project is a fork of [this fork](https://github.com/sofixa/nomad_follower) of this original [project](https://github.com/adragoset/nomad_follower).
Everything within is licensed under [GPL v3](./LICENSE)
25 changes: 15 additions & 10 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,32 +1,37 @@
module github.com/sas1024/nomad_follower
module github.com/Aiven-Open/nomad_logfetcher

go 1.14
go 1.24

require (
github.com/dmwilcox/fuzzytime v0.0.0-20191010161914-05ea0010feac
github.com/docker/go-units v0.4.0 // indirect
github.com/go-ldap/ldap v3.0.2+incompatible // indirect
github.com/hashicorp/nomad/api v0.0.0-20210702122822-4e9ed5f8827c
github.com/hashicorp/vault/api v1.1.0
github.com/mitchellh/mapstructure v1.4.1
gopkg.in/natefinch/lumberjack.v2 v2.0.0
)

require (
github.com/golang/snappy v0.0.4 // indirect
github.com/gorilla/websocket v1.4.2 // indirect
github.com/hashicorp/cronexpr v1.1.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
github.com/hashicorp/nomad/api v0.0.0-20210702122822-4e9ed5f8827c
github.com/hashicorp/vault/api v1.1.0
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/hashicorp/vault/sdk v0.2.0 // indirect
github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d // indirect
github.com/kr/pretty v0.2.1 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/go-testing-interface v1.14.1 // indirect
github.com/mitchellh/mapstructure v1.4.1
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/stretchr/testify v1.6.1 // indirect
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e // indirect
golang.org/x/net v0.0.0-20210614182718-04defd469f4e // indirect
golang.org/x/text v0.3.6 // indirect
golang.org/x/time v0.0.0-20210611083556-38a9dc6acbc6 // indirect
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/yaml.v2 v2.3.0 // indirect
)
Loading