# How does sincedb\_clean\_after in file{} work? **URL:** \ **Category:** Logstash\ **Created:** [December 19, 2018, 8:27am UTC](https://discuss.elastic.co/t/how-does-sincedb-clean-after-in-file-work/161471 "2018-12-19T08:27:36Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![espenk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/espenk/32/52199_2.png) [@espenk](https://discuss.elastic.co/u/espenk)\ **Post date:** [December 19, 2018, 8:27am UTC](https://discuss.elastic.co/t/how-does-sincedb-clean-after-in-file-work/161471/1 "2018-12-19T08:27:36Z")
Hi. I have been reading [https://www.elastic.co/guide/en/logstash/master/plugins-inputs-file.html](https://www.elastic.co/guide/en/logstash/master/plugins-inputs-file.html) for a while, and stumbled upon the following statement: "Sincedb records can now be expired meaning that read positions of older files will not be remembered after a certain time period." I'm aware that this points to the sincedb\_clean\_after option made available in March/April 2018. However, I seek to confirm that it actually works on my system. Is is expected that inode records gets deleted from sincedb, or does Logstash check if the inode entries have been expired at discovery interval? What is the expected behavior of this option? See the reply below, as to why I'm looking into this. ---
**Author:** ![espenk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/espenk/32/52199_2.png) [@espenk](https://discuss.elastic.co/u/espenk)\ **Post date:** [December 19, 2018, 6:01pm UTC](https://discuss.elastic.co/t/how-does-sincedb-clean-after-in-file-work/161471/2 "2018-12-19T18:01:15Z")
Background to why I'm looking into this, is that: I believe there still is a byte offset that is wrongly set in my test environment. Occasionally log lines are not properly read from the beginning. E.g "2018-12-24 host1 application2: did something" would be read into Logstash as "-24 host1 application2: did something". The folder which Logstash watches is subject to the following: - Rsyslog receivers data and saves as logtype.YYYY-MM-DD-HH:MM.log - Files increase in size over time, and are generated each minute. - Size of files are roughly 500 MB at working hours, 100 MB at night. - Files get gzipped if they are older than 6 hours - .gz files are excluded - If size of folder exceeds 40% of partitions size, remove files During my testing the files that had this behavior were still available, and I confirmed that the log line looks fine at disk. In other words, the files which were faulty read **were not deleted or gzipped**. File deletions are expected to occur roughly after 7 hours. Options set in file{}: sincedb\_clean\_after =\> 3.2 hours, ignore\_older =\> 3 hours. close\_inactive =\> 1 minute mode =\> tail exclude =\> .gz Logstash version: 6.5.2 CentOS 7 I have already verified that the logs do not contain any well-hidden 0x0a/0x0d chars that could be misinterpreted as newline. ---
**Author:** ![espenk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/espenk/32/52199_2.png) [@espenk](https://discuss.elastic.co/u/espenk)\ **Post date:** [December 21, 2018, 3:41pm UTC](https://discuss.elastic.co/t/how-does-sincedb-clean-after-in-file-work/161471/3 "2018-12-21T15:41:22Z")
Think I may have found the issue, as I have not had parse failures after. Increasing close\_inactive seem to be the fix for me. I wonder if it may be due to files being initially opened, but Logstash was unable to finish reading and closing the file handle within the expected 1 min. Thus, files were perhaps partially read? This seems like a likely cause, as files may stop increasing in size before logstash closes the file handle. Thus no size increase and won't be detected within ignore\_older. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2019, 3:41pm UTC](https://discuss.elastic.co/t/how-does-sincedb-clean-after-in-file-work/161471/4 "2019-01-18T15:41:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.