# Best practices to import millions of old log files to es

**URL:** <https://discuss.elastic.co/t/best-practices-to-import-millions-of-old-log-files-to-es/153089>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 19, 2018, 1:36am UTC](https://discuss.elastic.co/t/best-practices-to-import-millions-of-old-log-files-to-es/153089 "2018-10-19T01:36:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![guanghaofan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guanghaofan/32/31153_2.png) [@guanghaofan](https://discuss.elastic.co/u/guanghaofan)\
**Post date:** [October 19, 2018, 1:36am UTC](https://discuss.elastic.co/t/best-practices-to-import-millions-of-old-log-files-to-es/153089/1 "2018-10-19T01:36:50Z")

</div>

hi filebeat experts,

I'm going to import a huge amount of old files into ES, and my situation is there are millions of log files. So the main challenge is how to auto removed the log file once it's completely consumed by filebeat.

basically we can use the close\_eof, and clean\_removed to make sure all the logs files are send to ES smoothly. But how to know which file is fully consumed by filebeat since there's no a option something like remove/rename a file while EOF reached.

and now my way is either append a Event to each file and then read the es data to find the consumed files or reading the filebeat registry file/log file to get the consumed files.

So anybody who has the similar situation, please share your thoughts here.  
thanks.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 19, 2018, 1:57am UTC](https://discuss.elastic.co/t/best-practices-to-import-millions-of-old-log-files-to-es/153089/2 "2018-10-19T01:57:47Z")

</div>

Wy not just use cat + stdin to process the files, then delete once it has finished?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 16, 2018, 2:10am UTC](https://discuss.elastic.co/t/best-practices-to-import-millions-of-old-log-files-to-es/153089/3 "2018-11-16T02:10:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.