Read the same file several times

Elastic Stack Beats
1

Hi,

I have a file that I want to harvest many time with filebeat. That means the file has some logs (say 1000 logs lines) which don't change. Once filebeat has finish harvesting these 1000 logs, I want it to reharvest again those logs starting from the beginning.

I know that this can be done by deleting the file /var/lib/filebeat/registry but I don't want to do it that want. I was wondering if it is possible to configure filebeat so that it will reset all its offsets in the file /var/lib/filebeat/registry so that each time it will keep on sending the same logs.

2

You can use the clean_* options which will clean the registry. But to pick up the file again, the modtime must change. You could use touch for that. Can you share some more background on the above use case, especially what you use it for?

3

Thanks for the reply @ruflin. The idea is send some logs repeatly to a kafka topic. I will then collect those logs and do some analyses with them. Here is my config

filebeat.prospectors:

- paths:
    - "/path/to/logs/*.*"

output.kafka:
  enabled: true

  hosts: ["kakfaip1:9092","kakfaip2:9092","kakfaip3:9092"]

  topic: mytopic

  username: username
  password: pwduser

This config works well. But once filebeat have finished reading the files in /path/to/logs/, it stops sending logs to the topic mytopic. So I want filebeat to reread the file again and again starting from the begining each time.

4

Did you try out the clean_* options?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.