Filebeat Sending Whole File Repeatedly

Elastic Stack Beats
1

Hello everyone, so I have configure filebeat to send a its own configurations to kafka, the issue is that filebeat keeps shipping the configs every 10s even though there are no changes to these files. Is there any way to configure filbeat to ship a whole file as a message only when the the file has changed?

Config:
filebeat.prospectors:

- type: log

  paths:
    - ../config.yml

  multiline:
    pattern: ' \A.*'
    negate: true
    match: after
  fields_under_root: true
  fields:
    service: configs
    topic: <someTopic>
2

Filebeat is not really designed for this use case as the config is not a log file. As the size might be bigger or smaller after a change, you will have different behaviour after updating the config. There is not really a workaround here. One solution to this might be discussed here: https://github.com/elastic/beats/issues/3852

Taking a step back: What is the overall problem you are trying to solve?

3

We have about 80,000 agents deployed, and that number is increasing. The challenge we now have is how can we manage all the configurations so that we know whats out there and track any changes. The goal here was to have filebeat harvest its own configs as well as metricbeat's. Elastic search would then provide us the capability to query configs based on hostname and hash value.

4

Filebeat is designed to parse the appended content of a file. It's tracking the files it had worked with in its own registry file by saving the inode of the file and the offset value.
The offset value registers how many bytes from the beginning of the file have already been read before, so if a file became bigger, it continues with the byte after the saved offset value.
So if you change the content before the end (that is you are not appending but inserting/modifying data), filebeat won't be able to handle it.

You might want to check SVN or GIT out.

5

The issue I have is that even though there are no changes to the file filebeat keeps shipping it. And for other reasons we cant set up a git or svn repo with autosync and file acl in the repository.

6

You might be interested in: https://github.com/elastic/beats/issues/464

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.