# Logstash don't detect duplicated documents

**URL:** <https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543>\
**Category:** Logstash\
**Created:** [June 5, 2018, 7:13am UTC](https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543 "2018-06-05T07:13:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrigui94](https://avatars.discourse-cdn.com/v4/letter/c/a8b319/32.png) [@chrigui94](https://discuss.elastic.co/u/chrigui94)\
**Post date:** [June 5, 2018, 7:13am UTC](https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543/1 "2018-06-05T07:13:10Z")

</div>

I have to index dynamically multiple .json file in different folders but if I add the same json file that exists already logstash + Elasticsearch index it as a new file with the same documents. I need to find a solution in order to detect exited document and don't add them.  
Here is my input block:  
input {  
file {  
type =\> "solver"  
path =\> ["D:/Users/G361164/Desktop/test\_logstash0406/urban/indoor/\*.json"]  
start\_position =\> beginning  
# to read from the beginning of file  
sincedb\_path =\> "/dev/null"  
codec =\> "json"  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 5, 2018, 8:56pm UTC](https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543/2 "2018-06-05T20:56:25Z")

</div>

Logstash doesn't detect duplicates but if you configure it to explicitly set the document id when indexing to ES (rather than having ES pick a random document id) you can make sure that the second time a document is indexed it'll just overwrite the old (identical) document. You can use a fingerprint filter to compute the document id based on the fields and their values, then reference the new field containing the hash with the `document_id` option for your elasticsearch output.

This isn't the first time this comes up here so you should be able to find details in the archives.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 3, 2018, 8:58pm UTC](https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543/3 "2018-07-03T20:58:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.