# Filebeat data loss with file rotation and elasticsearch not reachable **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 17, 2017, 12:21pm UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238 "2017-10-17T12:21:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 17, 2017, 12:21pm UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238/1 "2017-10-17T12:21:22Z")
I am seeing loss of data in below scenario : - filebeat is sending data to elasticsearch from a json file - elasticsearch is down - multiple ( say 5 ) files are created during this time due to file rotation policy. files are named like below : log.json ( actual input file for filebeat ) arch\_2017-10-17.1.json ( format : arch\_{date}.{number}.json ) arch\_2017-10-17.2.json arch\_2017-10-17.3.json arch\_2017-10-17.4.json When I restarted elasticsearch, and checked Kibana I could only see the entries from log.json and arch\_2017-10-17.1.json. Entries from rest of the files are missing. Filebeat.yml : filebeat.prospectors: - input\_type: log output.elasticsearch: hosts: ["localhost:9200"] index: "elastbeat-%{+yyyy.MM.dd}" pipeline: "filebeat\_pipeline" bulk\_max\_size: 2048 processors: - drop\_fields: fields: ["beat.version","input\_type","offset", "tags"] What may be reason for the entries not seen in Kibana? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [October 17, 2017, 10:23pm UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238/2 "2017-10-17T22:23:57Z")
If you change the `paths` to also match the rotated file names then on restart Filebeat will detect that the file has been rotated and resume from the last offset. You can add a second path like `'C:\Users\arch*.json'` and that should resolve the issue. ---
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 18, 2017, 4:27am UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238/3 "2017-10-18T04:27:20Z")
I will try this out. I was just wondering if this would result in duplicate events being sent to elasticsearch. ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [October 18, 2017, 9:29am UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238/4 "2017-10-18T09:29:32Z")
It should not result in duplicate events. Filebeat is designed to handle log rotation. It knows that the file is the same even though the name has changed because it follows the underlying inode information. ---
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 18, 2017, 9:33am UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238/5 "2017-10-18T09:33:10Z")
@andrewkroh Thanks for the suggestion. I have tried including the arch-\* glob in the path and I can see the logs are published to elasticsearch. Haven't observed any duplicate entries so far. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 15, 2017, 9:33am UTC](https://discuss.elastic.co/t/filebeat-data-loss-with-file-rotation-and-elasticsearch-not-reachable/104238/6 "2017-11-15T09:33:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.