# Duplicate Issue - document\_id, how to prevent overwriting of entries

**URL:** <https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404>\
**Category:** Logstash\
**Created:** [February 16, 2021, 10:09am UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404 "2021-02-16T10:09:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sampson\_Light](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sampson_light/32/77283_2.png) [@Sampson\_Light](https://discuss.elastic.co/u/Sampson_Light)\
**Post date:** [February 16, 2021, 10:09am UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/1 "2021-02-16T10:09:32Z")

</div>

Hi All,

Some background information:

I have duplicate entries in my elasticsearch indexes.

Have used document\_id which prevented duplicates from appearing. But the issue with this is that it overwrites and updates the duplicate \> effectively removing the older copy which is 'correct'.

Can anyone point out if there is anyway to prevent the overwriting/updating from happening, and instead just tell ElasticSearch to ignore the duplicate that was detected from document\_id?

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [February 16, 2021, 10:44am UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/2 "2021-02-16T10:44:07Z")

</div>

Hi,

Elasticsearch has a parameter called `op_type`: [Index API | Elasticsearch Reference [7.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html)

Set this parameter to `create` to ingest only documents if the Id does not exist.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Sampson\_Light](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sampson_light/32/77283_2.png) [@Sampson\_Light](https://discuss.elastic.co/u/Sampson_Light)\
**Post date:** [February 16, 2021, 12:10pm UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/3 "2021-02-16T12:10:38Z")

</div>

Hi Wolfram,

Appreciate your response.

If you don't mind could you point out to me how I can apply your recommendation?

For example, am I supposed to edit a certain config file from somewhere? Or am I suppose to do something in the same logstash.conf file.

Regards  
Sam

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [February 16, 2021, 12:25pm UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/4 "2021-02-16T12:25:04Z")

</div>

Hello Sam,

This depends how you are ingesting the data:  
When ingesting directly to ElasticSearch `op_type` is an url parameter:

```auto
PUT my-index-000001/_doc/1?op_type=create

```

When using LogStash you can configure it in the [elasticsearch output](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action):

```auto
 output {
      elasticsearch {
        action => "create"
      }
    }

```

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Sampson\_Light](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sampson_light/32/77283_2.png) [@Sampson\_Light](https://discuss.elastic.co/u/Sampson_Light)\
**Post date:** [February 16, 2021, 1:35pm UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/5 "2021-02-16T13:35:58Z")

</div>

Hi Wolfram,

Apologies as I have forgotten to mention how I am using the data.

Yes, I am ingesting in via Logstash:

From the example you shown me, all I need to do is to add the actions =\> "create" line into my conf file.

After which Logstash will not index and overwrite documents which already have an ID \> if I were to simulate a duplicate this time round, nothing will happen (unlike before whereby documents were overwritten).

Am interpreting it right?

Regards  
Sam

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [February 16, 2021, 1:44pm UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/6 "2021-02-16T13:44:57Z")

</div>

Hello Sam,

Yes, adding `action => "create"` should solve your issue.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2021, 1:45pm UTC](https://discuss.elastic.co/t/duplicate-issue-document-id-how-to-prevent-overwriting-of-entries/264404/7 "2021-03-16T13:45:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.