Ignore_order not working in Filebeat

Elastic Stack Beats
1

In the filebeat.yml file we have ignore_order : 24h , still when filebeat was started for first time it picked up log events from all files which were present in the directory and matched with the naming pattern , some of which were having file timestamp older than 3 months from current date. Why is ignore_older not working and what is the way to ensure that filebeat picks up only certain files based on timestamp and ignores other files even if they are present in the directory ?

2

hi @PRODIPTO,

can you provide us with the full configuration, might be easier to spot the issue.

3

Please find the filebeat.yml file . The filebeat output is being fed into logstash for ingestion into elastic.

#=========================== Filebeat inputs =============================
filebeat.inputs:

- type: log
  enabled: true
  paths:
     - /xxxxxxx/xxxxx/xxxxxxxxxx/xxxxxx/xxxxx/yyyyyy_log.*
  fields:
     logtype: xxxxxx
     appname: abcd
     environment: xyzwpqrs
     ignore_older: 24h

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["xx.xx.xx.xx:abcd","xx.xx.xx.yy:abcd"]
  loadbalance: true
  worker: 2

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
4

Is there any change that needs to be made in the below filebeat.yml file to make the ignore_older clause work ?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.