File beat - handle too big registry file

Elastic Stack Beats
1

Hi Team,
I would like to understand how to handle filebeat too big registry file when logs files needs to be process only once (old logs(logs that are already in my logs folder, before my filebeat service will be in use) and new logs) no new lines will be written to file once he arrives in my logs folder.
I thought about one time solution of create for once the service without ignore_older and clean_inactive, let all the old logs to be processed with filebeat, disable the service and create it again with ignore_older and clean_inactive.
There are an alternative? or maybe I just don't understand filebeat well?

Thanks,
Sean Kotler

2

Any ideas?

3

What are settings in filebeat.yml for:

  • ignore_older
  • clean_inactive
  • scan_frequency

Have you read this?

4

Yes I do,
my problem is that I have old logs files that also needs to be process at the first run.
So if I will set ignore_older the log files wont be processed

5

Well if FB read logs older than X days/hours, no need to send again to ES.
Try to find your answer here, there is more details.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.