# Filebeat lost data **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 23, 2017, 12:03pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631 "2017-06-23T12:03:44Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\ **Post date:** [June 23, 2017, 12:03pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/1 "2017-06-23T12:03:44Z")
Hi all, We have experienced a lost data issue with filebeat \> logstash. Our Logstash server was down for 4 hours and some events have not been indexed. The filebeat configuration is as follows : ``` scan_frequency: 30s ignore_older: 10m close_eof: true close_removed: true clean_removed: true clean_inactive: 15m ``` The indexed files are read only once because they are never updated once created. I have identified some files that have not been indexed but I do not see them in filebeat logs (whereas I can see all other files). What is the exact retry policy of filebeat ? Can our configuration lead to data loss ? Thank you. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 23, 2017, 12:49pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/2 "2017-06-23T12:49:51Z")
filebeat uses send-at-least once semantics. Have you check the files state in the registry? A file might not be picked up if it has been deleted via log-rotation and filebeat is restart/started thereafter, or filebeat has closed a file and can not pick it up anymore, due to ignore\_older. ---
**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\ **Post date:** [June 23, 2017, 1:12pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/3 "2017-06-23T13:12:43Z")
What do you mean by send-at-least once sementics ? The "missing" files are not in registry. It seems that filebeat did not see these files, no trace in log file nor in the registry. Could it be due to broken communication with logstash ? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 26, 2017, 10:43am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/4 "2017-06-26T10:43:09Z")
> What do you mean by send-at-least once sementics ? That is, on failure (e.g. missing ACK from LS), filebeat will retry -\> harvesters will be blocked, due to buffers in filebeat being filled up. > It seems that filebeat did not see these files, no trace in log file nor in the registry. > > Could it be due to broken communication with logstash ? Maybe you want to share the full configuration, logs, registry file with us? Given the information I have so far, I'd assume it's due to `ignore_older` plus `clean_inactive`. The `clean_inactive` removes entries from the registry. Due to `ignore_older`, these old files are not picked up again... ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 26, 2017, 10:46am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/5 "2017-06-26T10:46:03Z")
What version of filebeat are we talking about? ---
**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\ **Post date:** [June 26, 2017, 4:01pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/6 "2017-06-26T16:01:14Z")
So when filebeat is in retry mode, harvesters end to be blocked due to buffer size. And if this situation lasts long (longer than `clean_inactive`, which should be longer than `ignore_older`), new files can be simply ignored. Is that a good summary of the situation? @ruflin I am using filebeat 5.4.1 ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 27, 2017, 8:22am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/7 "2017-06-27T08:22:14Z")
Not sure it's that simple, as filebeat behavior also depends on some other settings in your configuration file. Please share filebeat configuration. ---
**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\ **Post date:** [June 27, 2017, 8:52am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/8 "2017-06-27T08:52:27Z")
Here is the configuration, but there is nothing more than what I have previously shared : ``` filebeat.prospectors: - input_type: log paths: - D:\_data\aaa\bbb\*.csv encoding: utf-8 document_type: foo scan_frequency: 30s ignore_older: 10m close_eof: true close_removed: true clean_removed: true clean_inactive: 15m output.logstash: hosts: ["logstash-val:5044"] logging.level: info logging.to_files: true logging.files: path: D:\logs name: filebeat rotateeverybytes: 10485760 # = 10MB keepfiles: 10 ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 30, 2017, 8:46am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/9 "2017-06-30T08:46:04Z")
Could you try to set `close_removed` to false? ---
**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\ **Post date:** [June 30, 2017, 2:05pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/10 "2017-06-30T14:05:17Z")
The problem is I cannot easily test this scenario again. It occurred on a production environment while our logstash server was down for patching activity. It is a hard task to reproduce the environment in validation stage. Why do you think that `close_removed` can change something ? Missing files have not been removed. I think @steffens explanations are correct. Harvesters have been blocked for too long time and as `ignore_older` is quite short (10'), new files have been ignored. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [July 27, 2017, 11:30am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/12 "2017-07-27T11:30:52Z")
Sorry for the really late reply this somehow slipped through the cracks. You are right in case the files are not removed `close_removed` would not have an affect. As you haven't set a `harvester_limit` I would expect the harvester still pick up the new files preventing from applying ignore\_older. I wonder now if `close_inactive`could apply, close the file and the `clean_inactive` can happen. I would be really interesting to see the log file when this happens as this would allow to see the steps and logics that filebeat applied. ---
**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\ **Post date:** [July 27, 2017, 2:57pm UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/13 "2017-07-27T14:57:04Z")
No problem ! Thank you for answering. As I said, it is not an easy thing to simulate this behavior again. There is no patching activity scheduled on our logstash server (and I cannot stop it ! 😉) so just have to wait... I have change the `ignore_older` config to `2h` as I think this parameter comes into play in this issue. But maybe I'm wrong... If the issue happens again, I will send you the logs ! ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [July 31, 2017, 4:24am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/14 "2017-07-31T04:24:15Z")
@oguachard Thanks a lot, appreciate it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2017, 4:24am UTC](https://discuss.elastic.co/t/filebeat-lost-data/90631/15 "2017-08-28T04:24:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.