Count of files in sincedb and file_completed_log_path

Elastic Stack Logstash
1

Hi Can You explain why I have a so much difference in count of particular daily file in sincedb and file_complete_log_path. It seems that sincedb does not reflect the actual number of reprocessed files, hence there are no entries of these files in the records of the sincedb file, and every day it dumps excess data into elastica for me. Sometimes files of the same name appear in the directory and it does not filter them properly because of this.
docker.elastic.co/logstash/logstash:8.2.1

[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_20221001 | wc -l
387
[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_2022105 | wc -l
0
[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_20221005 | wc -l
562
[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_20221004 | wc -l
561
[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_20221003 | wc -l
508
[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_20221002 | wc -l
279
[elastic_user@host_b03 sincedb]$
[elastic_user@host_b03 sincedb]$ cat .sincedb | grep file_20221001 | wc -l
387
[elastic_user@host_b03 sincedb]$ cd ..
[elastic_user@host_b03 logstash7]$ cat fin_eir.log | grep file_20221001 | wc -l
545
[elastic_user@host_b03 logstash7]$ cat fin_eir.log | grep file_20221002 | wc -l
544
[elastic_user@host_b03 logstash7]$ cat fin_eir.log | grep file_20221003 | wc -l
542
[elastic_user@host_b03 logstash7]$ cat fin_eir.log | grep file_20221004 | wc -l
545
[elastic_user@host_b03 logstash7]$ cat fin_eir.log | grep file_20221005 | wc -l
545

2

Could be inode reuse. See this thread. There is no good solution.

3

Or should I add the parameter sincedb_write_interval to low number, because from default we have 15 sec.

4

I do not think that will fix it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.