[FILEBEAT] Insert timestamp inside of logfile name

Elastic Stack Beats
1

Hello,

I want to gather logs from a specific location. As the log's name is changed each day, I would like to insert timestamp in log file name. By doing so, each new day, the correct log name will be parsed by filebeat.

Log file name examples:

repository.cleanup-*20200921*010000083.log
repository.cleanup-*20200922*010000095.log
repository.cleanup-*20200923*010000097.log
repository.cleanup-*20200924*010000067.log

The text between stars in the file name I need to replace with a custom timestamp. The rest of the logfile name will be done by regex.

This is my input section:

- type: log
  enabled: true
  paths:
    - /opt/sonatype-work/nexus3/log/tasks/repository.cleanup-{SOME TIMESTAMP HERE}01{SOME REGEX}.log
  fields:
    kafka_topic: nexus-repository-cleanup-logs

Would be grateful for any suggestions!
Thank you.

2

Why do you need a special regex for the timestamp? Why not just use /opt/sonatype-work/nexus3/log/tasks/repository.cleanup-[0-9]{8}01[0-9]{6}.log?

3

Hello Noemi!

Well, the directory is full of old logs. And when I use a regex that can match any character like [0-9]{8}01[0-9]{6}, filebeat will go through all old logs and send their data to Kafka and so own to Elastic ( as old logs are not being zipped ).

So, to evade that and a sideway solution which would be to delete all old logs so it can start with the current active log, I have asked is it possible to use a timestamp in log name?

Also, another thing is, that, when logs get deleted from the Kafka topic after 7 days of retention, filebeat will instantly again parse those old logs and send them again to Kafka.

And when logs get deleted from the index in Elastic after some retention period, Kafka would also send those old logs again to Elastic. That will repeat constantly unless we setup some cron job that will delete logs on the server. But all of this is a not proper solution.

And in my opinion, best would be the timestamp in logfile name? Is that possible, and if so, do you have a suggestion?

Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.