Does Filebeat "churn" if input files are missing?

Elastic Stack Beats
1

If I have specified a number of logfiles from which to harvest entries, something like this:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/messages
      - /var/log/secure
      - /var/log/syslog
      - /opt/someapp/logs/audit.log
      - /opt/someapp/logs/debug.log
      - /opt/otherapp/logs/audit.log
      - /opt/otherapp/logs/debug.log

...and, ultimately, no otherapp is actually running on the host Filebeat is installed on, am I paying a penalty for this? I.e.: will Filebeat churn (waste cycles) looking over and over again for logs on the (unused) path /opt/otherapp/logs/ because I listed it? Is it a poor practice to "over-specify" logfile source paths?

I ask this because I want to (be lazy and) avoid having to write multiple versions of filebeat.yml and be careful about which host I install which version on. In my case, I'll really have at least three potentially different versions of filebeat.yml, maybe more.

2

@RussellBateman It wont affect performance, we only look periodically at the presence of files matching any defined path and this is done in a separate routine. Also from what I see you could use recursive glob to define path.

A list of glob-based paths that will be crawled and fetched. All patterns supported by Go Glob are also supported here. For example, to fetch all files from a predefined level of subdirectories, the following pattern can be used: /var/log/*/*.log. This fetches all .log files from the subfolders of /var/log. It does not fetch log files from the /var/log folder itself. It is possible to recursively fetch all files in all subdirectories of a directory using the optional recursive_glob settings.

Filebeat starts a harvester for each file that it finds under the specified paths. You can specify one path per line. Each line begins with a dash (-).

https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html#input-paths

1 Like
3

Yeah, I wrote the example verbosely in order to make it really clear what I was talking about. Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.