# Ignore\_older is not pushing the latest logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 2, 2020, 12:39pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632 "2020-07-02T12:39:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![aolvikash](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@aolvikash](https://discuss.elastic.co/u/aolvikash)\ **Post date:** [July 2, 2020, 12:39pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/1 "2020-07-02T12:39:28Z")
Hi All, I have configured ignore\_older into filebeat [version 6.6] so that only new logs should push into ELK, here is the ignore\_older and other related configs: > ignore\_older: 1h > close\_older: 24h > close\_inactive: 30m Keep getting the below logs till the log files are not older [\<1h]: > 2020-07-02T04:14:48.931-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: , offset: 16 > 2020-07-02T04:14:48.931-0700 DEBUG [input] log/input.go:548 File didn't change: Once the logs are older then started getting below logs: > 2020-07-02T04:23:51.674-0700 DEBUG [input] log/input.go:555 Ignore file because ignore\_older reached: The logs are not moving to ELK even after logs modified time is reached \> ignore\_older time. Could you please help here? ---
**Author:** ![aolvikash](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@aolvikash](https://discuss.elastic.co/u/aolvikash)\ **Post date:** [July 7, 2020, 2:08pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/2 "2020-07-07T14:08:37Z")
OR, Can someone please assit me on how to configure the filebeat to push the last 24 hours created logs only, I have huge logs but needs to push only last 24 hours created logs. Thanks in advance. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [July 7, 2020, 6:51pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/3 "2020-07-07T18:51:07Z")
AFAIK, [`close_older` is not a valid setting](https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-input-log.html#filebeat-input-log-close-options), so you can remove it. Other than that, your configuration looks good for what you are trying to achieve. Can you please post the following: - your entire `filebeat.yml` configuration (with any sensitive information redacted)? - the first minute or so of logs right after you start up Filebeat, with debug level logging turned on (which you seem to have done already!) - the output of `ls -l $YOUR_LOGS_FOLDER` where Filebeat is harvesting log files from. Thanks, Shaunak ---
**Author:** ![aolvikash](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@aolvikash](https://discuss.elastic.co/u/aolvikash)\ **Post date:** [July 8, 2020, 12:40pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/4 "2020-07-08T12:40:10Z")
Thanks, Shaunak for your time! Here are the requested details: - filebeat.yml: > #=========================== Filebeat inputs ============================= > > filebeat.inputs: > > - type: log > enabled: true > paths: > - /var/log/\*\* ## for test > ignore\_older: 24h > > #============================= Filebeat modules =============================== > > filebeat.config.modules: > path: ${path.config}/modules.d/\*.yml > reload.enabled: false > > #==================== Elasticsearch template setting ========================== > setup.template.name: "" > setup.template.pattern: "\" > setup.template.settings: > index.number\_of\_shards: 3 > > #============================== Kibana ===================================== > > setup.kibana: > > host: "localhost:5601" > #================================ Outputs ===================================== > > #-------------------------- Elasticsearch output ------------------------------ > output.elasticsearch: > hosts: ["localhost:9200"] > protocol: "https" > username: "" > password: "" > pipeline: > > #================================ Logging ===================================== > > logging.level: debug > logging.metrics.period: 600s > logging.to\_files: true > logging.files: > path: > name: filebeat > rotateeverybytes: 5055760 > keepfiles: 4 > permissions: 0644 - filebeat logs: > 2020-07-08T05:31:20.758-0700 DEBUG [input] input/input.go:152 Run input > 2020-07-08T05:31:20.758-0700 DEBUG [input] log/input.go:174 Start next scan > 2020-07-08T05:31:20.758-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/ > 2020-07-08T05:31:20.759-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/mnt > 2020-07-08T05:31:20.759-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/mnt/tmp > 2020-07-08T05:31:20.760-0700 DEBUG [input] log/input.go:267 Skipping directory: /var/log/mnt/tmp/p8dump > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/db\_stats.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:555 Ignore file because ignore\_older reached: /var/log/mnt/tmp/p8dump/db\_stats.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/db\_test.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:555 Ignore file because ignore\_older reached: /var/log//mnt/tmp/p8dump/db\_test.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/test1.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:555 Ignore file because ignore\_older reached: /var/log/mnt/tmp/p8dump/test1.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/test\_1m.txt > 2020-07-08T05:31:20.761-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /var/log/mnt/tmp/p8dump/test\_1m.txt, offset: 19 > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:548 File didn't change: /var/log/mnt/tmp/p8dump/test\_1m.txt > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/test\_2m.txt > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /var/log/mnt/tmp/p8dump/test\_2m.txt, offset: 22 > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:548 File didn't change: /var/log/mnt/tmp/p8dump/test\_2m.txt > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /var/log/mnt/tmp/p8dump/db\_env.txt > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:555 Ignore file because ignore\_older reached: /var/log/mnt/tmp/p8dump/db\_env.txt > 2020-07-08T05:31:20.762-0700 DEBUG [input] log/input.go:195 input states cleaned up. Before: 6, After: 6, Pending: 0 - output of `ls -l $YOUR_LOGS_FOLDER`: > cc1-saro-1# ls -ld /var/log/ > drwxr-xr-x 3 root wheel 3 Jul 2 01:32 /var/log/ > cc1-saro-1# ---
**Author:** ![aolvikash](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@aolvikash](https://discuss.elastic.co/u/aolvikash)\ **Post date:** [July 9, 2020, 3:08pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/5 "2020-07-09T15:08:18Z")
Hello Shaunak, Could you please provide your thoughts on the above issue. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [July 9, 2020, 6:51pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/6 "2020-07-09T18:51:30Z")
Is there a network-mounted folder inside `/var/logs`? Also what does `find /var/log -mtime -24h` return? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 6, 2020, 9:05pm UTC](https://discuss.elastic.co/t/ignore-older-is-not-pushing-the-latest-logs/239632/7 "2020-08-06T21:05:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.