Feature request: Parse multiple log files based on wildcard

[johnvk-dev]

I'm planning on using this tool to parse the logs of a Postfix server. The logs are rotated daily at 02:00. I do not want to miss any mail logs. Timing the parse job right before the rotation could mean I miss some mail logs.

A suggestion would be to parse multiple logs via a wildcard, so when we run the parse job, it will see the logs in the active log as well as the rotated one (we keep the active log and the last day log uncompressed). This assumes that duplicate logs are not imported and it would be safe (not duplicating data, not placing heavy load on the server) to run this parse job every 15 minutes.

a typical daily mail log is about 75 to 100 mb in size uncompressed.

There might be a better solution, thank you for considering my use case!

[drlight17]

Hm... I didn't deal with such heavyload mail servers so parser in my case is run every minute.
I'll check the ability to parse multiple files by wildcard name later

[johnvk-dev]

I appreciate your effort! I explored the options via 'Code with agent mode'. There were some good (imo, im not a developer by trade) ideas from that session

1. using python glob to collect the files matching a wildcard and parsing them in a loop
2. using a checkpoint based on the inode stats of the log file to only read the bytes not read after the last parse action. the checkpoint was stored in a local json file on the container

[drlight17]

2. using a checkpoint based on the inode stats of the log file to only read the bytes not read after the last parse action. the checkpoint was stored in a local json file on the container

Yeah, I've also had thoughts to optimize parser in the kind of this way.