# Ignore\_older concept is not working

**URL:** <https://discuss.elastic.co/t/ignore-older-concept-is-not-working/359531>\
**Category:** Elasticsearch\
**Created:** [May 15, 2024, 9:55am UTC](https://discuss.elastic.co/t/ignore-older-concept-is-not-working/359531 "2024-05-15T09:55:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![babudurairaji](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@babudurairaji](https://discuss.elastic.co/u/babudurairaji)\
**Post date:** [May 15, 2024, 9:55am UTC](https://discuss.elastic.co/t/ignore-older-concept-is-not-working/359531/1 "2024-05-15T09:55:29Z")

</div>

Hi,

ignore\_older concept is not working in our use case.

Please find the below our use case :

1. We took a backup of the Elasticsearch folder installed on your Windows server at 10 AM on May 15, 2024.
2. Elasticsearch (ES), Filebeat, and Logstash continued to work and push data to ES until 12 AM on May 15, 2024.
3. At 12 AM, we deleted the entire ES folder.
4. We restored the backup of ES that was taken at 10 AM on May 15, 2024.
5. After refreshing the backup ES, we could see the data saved until 10 AM on May 15, 2024.
6. The Filebeat log file contains monitor and harvested information until 12 PM.
7. we require to retrieve the missing data between 10 AM and 12 PM.
8. We set `ignore_older: 2h` in the Filebeat configuration (`filebeat.yml`).
9. However, after restarting Filebeat and Logstash, it is not pushing the missing data, and there are no errors in the log file; only monitor entries are present.

Can you please provide your suggestion on this issue?

shared the filebeat.yml file for your reference.

```auto
=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - C:\LIMSAudit\AuditTextFilePath\\specimen-*.json
  fields: {log_type: specimen}
  ignore_older: 2h
  
- type: log
  enabled: true
  paths:
  - C:\LIMSAudit\AuditTextFilePath\\useractivity-*.json
  fields: {log_type: useractivity}
  ignore_older: 2h

type: log
  enabled: true
  paths:
  - C:\LIMSAudit\AuditTextFilePath\\order-*.json
  fields: {log_type: order}
  ignore_older: 2h
  
- type: log
  enabled: true
  paths:
  - C:\LIMSAudit\AuditTextFilePath\\profile-*.json
  fields: {log_type: profile}
  ignore_older: 2h

```

[/quote]

---

<div class="post-metadata">

**Author:** ![babudurairaji](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@babudurairaji](https://discuss.elastic.co/u/babudurairaji)\
**Post date:** [May 17, 2024, 10:17am UTC](https://discuss.elastic.co/t/ignore-older-concept-is-not-working/359531/2 "2024-05-17T10:17:29Z")

</div>

Hi,

I am unsure why Filebeat is not considering the ignore\_older value. We need your guidance to solve this problem.

I appreciate any help you can provide.

Regards,  
Babu

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 17, 2024, 10:25am UTC](https://discuss.elastic.co/t/ignore-older-concept-is-not-working/359531/3 "2024-05-17T10:25:00Z")

</div>

While Filebeat is reading log files it keeps track of the position in a file on disk. It will not reprocess already read events while this is in place. Setting `ignore_older` does not make Filebeat reprocess files, it just makes it skips new files it identifies that were last modified earlier than the `ignore_older` threshold.

It therefore sounds like you may have misunderstood the purpose of this parameter.

---

<div class="post-metadata">

**Author:** ![babudurairaji](https://avatars.discourse-cdn.com/v4/letter/b/53a042/32.png) [@babudurairaji](https://discuss.elastic.co/u/babudurairaji)\
**Post date:** [May 17, 2024, 10:35am UTC](https://discuss.elastic.co/t/ignore-older-concept-is-not-working/359531/4 "2024-05-17T10:35:17Z")

</div>

Thank you for the information.  
Can you please advise us on how to restore the missing data between 10 AM and 12 PM?

Is there any configuration available for Filebeat? If Filebeat doesn't have any configuration, could you provide the best approach to handle this scenario?

Regards,  
Babu