# Parse the past - How to manage my log files **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 14, 2017, 9:29am UTC](https://discuss.elastic.co/t/parse-the-past-how-to-manage-my-log-files/78481 "2017-03-14T09:29:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Pierre\_Vincent\_Ledou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierre_vincent_ledou/32/1368_2.png) [@Pierre\_Vincent\_Ledou](https://discuss.elastic.co/u/Pierre_Vincent_Ledou)\ **Post date:** [March 14, 2017, 9:29am UTC](https://discuss.elastic.co/t/parse-the-past-how-to-manage-my-log-files/78481/1 "2017-03-14T09:29:30Z")
Hi, I have installed a ELK stack in production to get stats from our cdn logs. I have about 6 month of past logs to parse, with between 50 and 200 millions of event per day. Logs are stored in gz. Currently, I have a script that uncompresses the logs (40 per batch) in a directory watched by Filebeat. But I have no way to know when the 40 files have been parsed to start an other batch, so I'm doing by hand... Any idea how my script could know when filebeat as finished? I thought I could based on the registry, but there is no info about the fact that files have been read to the end. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 15, 2017, 10:59pm UTC](https://discuss.elastic.co/t/parse-the-past-how-to-manage-my-log-files/78481/2 "2017-03-15T22:59:53Z")
I would recommend you to use the registry. You can use the offset in the registry and compare it with the file size. If size == offset, filebeat is finished with reading. Otherwise have a look at the `-once` option, but that would mean to start filebeat every time. ---
**Author:** ![Pierre\_Vincent\_Ledou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierre_vincent_ledou/32/1368_2.png) [@Pierre\_Vincent\_Ledou](https://discuss.elastic.co/u/Pierre_Vincent_Ledou)\ **Post date:** [March 16, 2017, 9:35am UTC](https://discuss.elastic.co/t/parse-the-past-how-to-manage-my-log-files/78481/3 "2017-03-16T09:35:14Z")
Oh I didn't got that offset==file size, nice! I finally managed it by logging filebeat activity in a file, and parsing it to get the 'File is inactive: path\_to\_the/file'. So now I have a pretty nice log file manager script croned every 5 min. I uncompress 40 logs, move it to filebeat watch folder, and archive them when finished. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 13, 2017, 9:35am UTC](https://discuss.elastic.co/t/parse-the-past-how-to-manage-my-log-files/78481/4 "2017-04-13T09:35:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.