# Problem with sincedb **URL:** \ **Category:** Logstash\ **Created:** [April 24, 2017, 2:10pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420 "2017-04-24T14:10:23Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ktomu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktomu/32/3975_2.png) [@ktomu](https://discuss.elastic.co/u/ktomu)\ **Post date:** [April 24, 2017, 2:10pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420/1 "2017-04-24T14:10:24Z")
Hello, I'm using the file input with Logstash - just plane text files (pipeline processing) output to elasticsearch and without any filters. Sometimes I have an issue, when some files have processed but there are no new lines added to the sincedb until I have restarted the logstash. After restart the new lines just added to the sincedb. It hapens not for all files and not on the same condition. My script uses the sincedb to be aware which files are ready to remove - that's why It is very important to me. Can someone please explain, how can I solve this problem? Logstash & Elasticsearch - 5.2 ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [April 24, 2017, 3:56pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420/2 "2017-04-24T15:56:09Z")
I'm not completely clear on what your issue is. Are you saying that new lines are not being recognized by Logstash after a restart, but the sincedb is being updated anyway? ---
**Author:** ![ktomu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktomu/32/3975_2.png) [@ktomu](https://discuss.elastic.co/u/ktomu)\ **Post date:** [April 24, 2017, 8:35pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420/3 "2017-04-24T20:35:11Z")
Sorry, if a file has processed, Logstash reports about it in .sincedb by addding the new line with inode, offset ...of the file. But, sometimes, even if some files have already processed (i see the end of the files in a kafka topic - my output way) Logstash isn't reporting about it in .sincedb. It looks like Logstash being in stuck with updating the .sincedb. Only after restart Logstach is reporting about all processed files in sincedb. #processing file 1 - processed - reported file 2 - processed - reported file 3 - processed - reported file 4 - processed - but not reported file 5 - processed - but not reported file 6 - processed - but not reported file N - processed - but not reported #restart logstash reported about 4,5,6...N ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [April 24, 2017, 8:39pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420/4 "2017-04-24T20:39:42Z")
It is possible that Logstash is having this issue. Logstash is not a good fit for a "start up, read files, shut down" use case. In fact, we now recommend users use filebeat instead for file tailing. But filebeat isn't likely to be a good fit for what it looks like you are doing. The bottom line is that neither Logstash nor filebeat are meant to batch-read entire files and shut down until more files arrive. They are both designed to run continuously. I think you will be better suited with a custom script or tool for this use case. ---
**Author:** ![ktomu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ktomu/32/3975_2.png) [@ktomu](https://discuss.elastic.co/u/ktomu)\ **Post date:** [May 2, 2017, 7:11pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420/5 "2017-05-02T19:11:47Z")
Ok, thanks a lot ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 30, 2017, 7:25pm UTC](https://discuss.elastic.co/t/problem-with-sincedb/83420/6 "2017-05-30T19:25:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.