# How to stop duplicate entries using elasticsearch plugin **URL:** \ **Category:** Logstash\ **Created:** [June 1, 2017, 10:12am UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880 "2017-06-01T10:12:48Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Matthew\_Byrne](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@Matthew\_Byrne](https://discuss.elastic.co/u/Matthew_Byrne)\ **Post date:** [June 1, 2017, 10:12am UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/1 "2017-06-01T10:12:48Z")
Morning all. Im trying to use the elasticsearch plugin to pull data from an existing elasticsearch instance. Both are running older versions of Elasticsearch, and thus im running an older version of logstash (1.5). I've having 2 issues so far. 1. The logstash instance runs to a point, and then shuts down. (this isnt a major problem is problem 2 can be solved). 2. When I start up the logstash instance again, it copies over data which has already copied over, thus creating duplicate entries in my new elasticsearch instance. Input: `` ``` input { elasticsearch { hosts => ["kibana.host.name"] query => '{ "query": { "match": { "message": "filterMessageHere" } } }' docinfo => true scroll => '2m' } } ``` Output: `` ``` output { elasticsearch { host => localhost } } ``` I tried to add document\_id =\> "%{\_id}" (since thats the id defined in the source elasticsearch instance), but had no success. Any help would be much appreciated. Thanks all. ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [June 1, 2017, 11:53am UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/2 "2017-06-01T11:53:04Z")
If id, type and destination index of the documents are the same, by default it should not create another instance of the same document, but rather just bump the version number of the already indexed document. You can always try changing the default action of the output plugin to _create_ instead of the default _index_ (as per [https://www.elastic.co/guide/en/logstash/1.5/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action](https://www.elastic.co/guide/en/logstash/1.5/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action)), so the insert will fail for already existing documents. Keep in mind though that you will potentially be flooded with 40x error responces from ES. Nothing to worry about since it's intended, but may take up space quickly depending on the amount of them. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2017, 12:08pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/3 "2017-06-01T12:08:13Z")
If you look in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-docinfo) the example given under the `docinfo` section, this seems to show how to assign document id from the metadata fields, which is the default location for this information. ---
**Author:** ![Matthew\_Byrne](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@Matthew\_Byrne](https://discuss.elastic.co/u/Matthew_Byrne)\ **Post date:** [June 1, 2017, 1:11pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/4 "2017-06-01T13:11:31Z")
I had set the docinfo =\> true, but didn't see any improvement (unless I need to dump the indexes first and start copying again). @paz I tried setting the default action to create\_unless\_exists earlier, but no joy. I'll try 'create' now. You're right about the logs though, flooding with 40x error responses. Am I right in thinking that logstash is trying to copy over data that it already has, and is erroring out because it already exists? This would explain why my document counts aren't increasing (yet). Eventually i'd expect to see logstash find an index that wasnt copied over and start increasing my document count. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 1, 2017, 1:14pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/5 "2017-06-01T13:14:59Z")
Did you look in the documentation I linked to? The example shows how you set the document id: `document_id => "%{[@metadata][_id]}"` ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [June 1, 2017, 1:15pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/6 "2017-06-01T13:15:57Z")
> [@Matthew\_Byrne](#): > > ...Am I right in thinking that logstash is trying to copy over data that it already has, and is erroring out because it already exists? This would explain why my document counts aren't increasing (yet). Eventually i'd expect to see logstash find an index that wasnt copied over and start increasing my document count. That is correct, Elasticsearch refuses to create the document since it already exists, and the error propagates back to Logstash. When the scroll goes beyond the documents already indexed, you should see those errors stopping and the document count increasing. ---
**Author:** ![Matthew\_Byrne](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@Matthew\_Byrne](https://discuss.elastic.co/u/Matthew_Byrne)\ **Post date:** [June 1, 2017, 1:33pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/7 "2017-06-01T13:33:40Z")
@Christian_Dahlqvist I sure did Christian. I updated my input file to include docinfo =\> true and my output file as follows: ``` output { elasticsearch { host => localhost action => "create" index => "logstash-%{YYYY.MM.dd}" document_type => "%{[@metadata][_type]}" document_id => "%{[@metadata][_id]}" } } ``` Im running an older version of logstash so im looking at [this documentation](https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-docinfo). @paz & @Christian_Dahlqvist I deleted all my indexes and restarted logstash with the above output config, but im getting a constant stream of warns in the logs and no documents being indexed. `:message=>"failed action with response of 400, dropping action` ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [June 1, 2017, 1:37pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/8 "2017-06-01T13:37:45Z")
Can you post a sample from the Elasticsearch log? There should be more information there on why it returns 400. ---
**Author:** ![Matthew\_Byrne](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@Matthew\_Byrne](https://discuss.elastic.co/u/Matthew_Byrne)\ **Post date:** [June 1, 2017, 1:38pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/9 "2017-06-01T13:38:39Z")
I removed the index from my output configuration and the logs are all quiet and documents are being indexed. ---
**Author:** ![Matthew\_Byrne](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@Matthew\_Byrne](https://discuss.elastic.co/u/Matthew_Byrne)\ **Post date:** [June 1, 2017, 1:44pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/10 "2017-06-01T13:44:00Z")
Ah, I wonder was it the index naming that was breaking it: `{:timestamp=>"2017-06-01T13:35:22.377000+0000", :message=>"failed action with response of 400, dropping action: [\"index\", {:_id=>\"AVxUpH8GqqYqcknYGhjV\", :_index=>\"logstash-%{YYYY.MM.dd}\",`\ logstash-%{YYYY.MM.dd} wasn't getting translated. I'll let it index away for now, and circle back here when I restart logstash. Hopefully it won't duplicate the data this time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 29, 2017, 1:44pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880/11 "2017-06-29T13:44:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.