# Logstash -\> drop duplicate -\> elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [March 8, 2017, 11:42am UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807 "2017-03-08T11:42:54Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Exocomp](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Post date:** [March 8, 2017, 11:42am UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/1 "2017-03-08T11:42:54Z")
Hello, Looking over documentation I see that the field " **document\_id**" can be used as a way to upsert in Elasticsearch. However, is there way to detect the duplicate at the filter stage and just drop? Thanks, E ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 12, 2017, 7:38pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/2 "2017-03-12T19:38:19Z")
Theoretically you could probably use the elasticsearch filter but that would be insanely inefficient. ---
**Author:** ![Exocomp](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Post date:** [March 12, 2017, 11:24pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/3 "2017-03-12T23:24:27Z")
@magnusbaeck yea that would not be good. How about something like the following: 1. Local file that tracks all the document ids which gets populated at the output stage 2. At the filter stage you simple check against that file and if you find a matching document id you drop Side effects would be if elasticsearch output fails and file output succeeds then you don't get document in elasticsearch and think it succeeded. To resolve this side effect you would need a third step. 1. If elasticsearch output fails then catch the failure and remove the document id from the local file This would be a pretty efficient solution, based on your experience can you recommend the steps necessary to make this possible. Thanks, E ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 13, 2017, 7:58am UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/4 "2017-03-13T07:58:13Z")
> 1. Local file that tracks all the document ids which gets populated at the output stage 1. At the filter stage you simple check against that file and if you find a matching document id you drop Yes, that approach would work. Write the IDs to a file with a file output and use a translate filter to look up from that file. > 1. If elasticsearch output fails then catch the failure and remove the document id from the local file Sorry, that's not possible unless you modify the elasticsearch output. ---
**Author:** ![Exocomp](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Post date:** [March 13, 2017, 3:18pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/5 "2017-03-13T15:18:17Z")
@magnusbaeck thanks for your input. Regarding the 3rd step, I filed a feature request [https://github.com/elastic/logstash/issues/6814](https://github.com/elastic/logstash/issues/6814) Thanks, E ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2017, 3:23pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/6 "2017-03-13T15:23:44Z")
The problem with trying to detect duplicate entries in Logstash is that it basically requires all events (at least within a stream) to pass through a single instance, and therefore does not scale well. I think it is generally better to assign a unique id and let Elasticsearch handle it. ---
**Author:** ![Exocomp](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Post date:** [March 13, 2017, 3:42pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/7 "2017-03-13T15:42:38Z")
@Christian_Dahlqvist I see what your saying, so basically in order to detect a duplicate in logstash properly you'd have to create some kind of duplicate service and find a way to scale it which adds more complexity. Hmm. Ok, so as you said if Elasticsearch can handle it, basically a feature in elasticsearch to drop it would be more efficient. I will move my feature request from logstash to elasticsearch. Thanks, E ---
**Author:** ![Exocomp](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Post date:** [March 13, 2017, 4:10pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/8 "2017-03-13T16:10:15Z")
@magnusbaeck @Christian_Dahlqvist @theuntergeek mentioned that this option already exists with elasticsearch with the action create option. See here [https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/574](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/574) So if the document id already exists it will just fail, seems more efficient to me then doing the upsert, I'll go ahead with that option. Thanks for everyone's input. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 10, 2017, 4:10pm UTC](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/9 "2017-04-10T16:10:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.