If I set sincedb_path = /dev/null in a logstash configuration, that means that every time I run logstash, de data will be re-indexed in elastic?
this will create duplicates?
what problems could arise with this?
If sincedb_path is set to /dev/null then the in-memory sincedb is not persisted across restarts. So every time logstash restarts it will re-read the files. Unless you are setting the document_id based on the content of the event that will result in duplicates. This results in it re-doing work, which is wasteful, and it results in duplicate records in elasticsearch. Whether either of these is a problem is really up to you.
1 Like
I was seeing the confs of a fellow worker, and find out that in his configurations he has setted sincedb to null, the system has been restarted many times, so I guess that means that there is more than duplicate, and triplicates?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.