Ignore_older is not pushing the latest logs

Elastic Stack Beats
1

Hi All,

I have configured ignore_older into filebeat [version 6.6] so that only new logs should push into ELK, here is the ignore_older and other related configs:

ignore_older: 1h
close_older: 24h
close_inactive: 30m

Keep getting the below logs till the log files are not older [<1h]:

2020-07-02T04:14:48.931-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: , offset: 16
2020-07-02T04:14:48.931-0700 DEBUG [input] log/input.go:548 File didn't change:

Once the logs are older then started getting below logs:

2020-07-02T04:23:51.674-0700 DEBUG [input] log/input.go:555 Ignore file because ignore_older reached:

The logs are not moving to ELK even after logs modified time is reached > ignore_older time.

Could you please help here?

2

OR,

Can someone please assit me on how to configure the filebeat to push the last 24 hours created logs only, I have huge logs but needs to push only last 24 hours created logs.

Thanks in advance.

3

AFAIK, close_older is not a valid setting, so you can remove it. Other than that, your configuration looks good for what you are trying to achieve.

Can you please post the following:

  • your entire filebeat.yml configuration (with any sensitive information redacted)?
  • the first minute or so of logs right after you start up Filebeat, with debug level logging turned on (which you seem to have done already!)
  • the output of ls -l $YOUR_LOGS_FOLDER where Filebeat is harvesting log files from.

Thanks,

Shaunak

4

Thanks, Shaunak for your time!

Here are the requested details:

  • filebeat.yml:

#=========================== Filebeat inputs =============================

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /var/log/** ## for test
      ignore_older: 24h

#============================= Filebeat modules ===============================

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

#==================== Elasticsearch template setting ==========================
setup.template.name: ""
setup.template.pattern: "<pipelinename-*>"
setup.template.settings:
index.number_of_shards: 3

#============================== Kibana =====================================

setup.kibana:

host: "localhost:5601"
#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["localhost:9200"]
protocol: "https"
username: ""
password: ""
pipeline:

#================================ Logging =====================================

logging.level: debug
logging.metrics.period: 600s
logging.to_files: true
logging.files:
path:
name: filebeat
rotateeverybytes: 5055760
keepfiles: 4
permissions: 0644

  • filebeat logs:

2020-07-08T05:31:20.758-0700 DEBUG [input] input/input.go:152 Run input
2020-07-08T05:31:20.758-0700 DEBUG [input] log/input.go:174 Start next scan
2020-07-08T05:31:20.758-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/
2020-07-08T05:31:20.759-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/mnt
2020-07-08T05:31:20.759-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/mnt/tmp
2020-07-08T05:31:20.760-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/mnt/tmp/p8dump
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/db_stats.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:555 Ignore file because ignore_older reached: /var/log/mnt/tmp/p8dump/db_stats.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/db_test.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:555 Ignore file because ignore_older reached: /var/log//mnt/tmp/p8dump/db_test.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/test1.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:555 Ignore file because ignore_older reached: /var/log/mnt/tmp/p8dump/test1.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/test_1m.txt
2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /var/log/mnt/tmp/p8dump/test_1m.txt, offset: 19
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:548 File didn't change: /var/log/mnt/tmp/p8dump/test_1m.txt
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/test_2m.txt
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /var/log/mnt/tmp/p8dump/test_2m.txt, offset: 22
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:548 File didn't change: /var/log/mnt/tmp/p8dump/test_2m.txt
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/db_env.txt
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:555 Ignore file because ignore_older reached: /var/log/mnt/tmp/p8dump/db_env.txt
2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:195 input states cleaned up. Before: 6, After: 6, Pending: 0

  • output of ls -l $YOUR_LOGS_FOLDER:

cc1-saro-1# ls -ld /var/log/
drwxr-xr-x 3 root wheel 3 Jul 2 01:32 /var/log/
cc1-saro-1#

5

Hello Shaunak,

Could you please provide your thoughts on the above issue.

6

Is there a network-mounted folder inside /var/logs? Also what does find /var/log -mtime -24h return?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.