Ignoring old files

Elastic Stack Beats
1

I'm new to beats. I am having a lot of trouble getting filebeat to ignore old files. I have ignore_older: 48h in my filebeats.yml but it does not seem to be working. Why? My folder contains millions of files and it's important that filebeat doesn't try to work on them all.

Can anyone point me in the right direction? What is wrong with my config? Copy below:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/myfolders/log/*/*.log
scan_frequency: 60s
ignore_older: 48h
close_inactive: 72h
compression_level: 0
close_renamed: true
close_removed: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
logging.to_syslog: true
output.logstash:
  hosts: ["10.10.10.10:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Any tips for a beats rookie would be appreciated!

2

The indentation seems to be incorrect. scan_frequency, ignore_older, etc are input level options, so they need to be indented accordingly.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/myfolders/log/*/*.log
  scan_frequency: 60s
  ignore_older: 48h
  close_inactive: 72h
  compression_level: 0
  close_renamed: true
  close_removed: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
logging.to_syslog: true
output.logstash:
  hosts: ["10.10.10.10:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.