# Need own unqiue ID with Bulk insert

**URL:** <https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599>\
**Category:** Elasticsearch\
**Created:** [June 3, 2020, 4:05pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599 "2020-06-03T16:05:43Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![elasticitm](https://avatars.discourse-cdn.com/v4/letter/e/9fc29f/32.png) [@elasticitm](https://discuss.elastic.co/u/elasticitm)\
**Post date:** [June 3, 2020, 4:05pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/1 "2020-06-03T16:05:44Z")

</div>

Hallo @all,

I've switched from a older elasticsearch version to the current 7.7 and now getting troubles using unique id's.

In the newer elasticsearch version the "\_id" is set automatically to a short one and this breaks the whole logic of my implementation.

My product id (uuid) is for example: 709\_dis\_\_29618840141927\_252041531

if I import products in bulk and the id exits the product should be overwritte for example:

```auto
PUT {{base_url}}/_bulk
{"index":{"_index":"{{domain}}_product"}, "_id": "709_dis__29618840141927_252041531",}
{"date": "2019-01-01", "price": 200, "promoted": true, "rating": 1, "type": "hat"}

```

I know:

- "better performance with shorter id" - **it is not an option**!
- "using hash to shorten" - **it is not an option**!

Is there a work arrount for setting own unique id with my need length (if ES is slower in this case thats finde for me)?

Thank you for your help!

---

<div class="post-metadata">

**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\
**Post date:** [June 3, 2020, 4:17pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/2 "2020-06-03T16:17:50Z")

</div>

@elasticitm Hey, maybe the fingerprint filter / processor could be an option for you? Check this article to get an idea =\> [https://www.elastic.co/blog/logstash-lessons-handling-duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)

Grtz

---

<div class="post-metadata">

**Author:** ![elasticitm](https://avatars.discourse-cdn.com/v4/letter/e/9fc29f/32.png) [@elasticitm](https://discuss.elastic.co/u/elasticitm)\
**Post date:** [June 3, 2020, 5:08pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/3 "2020-06-03T17:08:36Z")

</div>

Hi Willem,

thanks for your replay! Properly it could solve the problem, but makes the system more compley, so I would like to avoid using locklash.

I found out, that it works very fine when I add a single product instead of bulk:

```auto
PUT {{base_url}}/{{domain}}_product/_doc/709_dis__29618840141927_252041531

```

Question is now if there is a working syntax for bulk insert updates, that allows to set with own "\_id"?

Best Regards

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 3, 2020, 5:14pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/4 "2020-06-03T17:14:08Z")

</div>

Can you try setting `_type` to `_doc` in the bulk request and see if that changes anything?

---

<div class="post-metadata">

**Author:** ![elasticitm](https://avatars.discourse-cdn.com/v4/letter/e/9fc29f/32.png) [@elasticitm](https://discuss.elastic.co/u/elasticitm)\
**Post date:** [June 3, 2020, 5:37pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/5 "2020-06-03T17:37:19Z")

</div>

Hi Christian,

thanks for this hint 🤙

setting \_doc like this

```auto
PUT {{base_url}}/_doc/_bulk

```

brings no effect

I'm not sure how I can set type here. Could you help me with the code?

Best Regards

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 3, 2020, 5:41pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/6 "2020-06-03T17:41:14Z")

</div>

I meant putting it in each bulk header next to the \_id.

---

<div class="post-metadata">

**Author:** ![elasticitm](https://avatars.discourse-cdn.com/v4/letter/e/9fc29f/32.png) [@elasticitm](https://discuss.elastic.co/u/elasticitm)\
**Post date:** [June 4, 2020, 11:47am UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/7 "2020-06-04T11:47:01Z")

</div>

Hi, Christian,

unforunetly that doesn't fix the problem.

Here is my request

```auto
PUT {{base_url}}/_bulk
{"index":{"_index":"{{domain}}_product"}, "_id": "709_dis__29618840141927_252041531", "_type":"_doc"}
{"id": "709_dis__29618840141927_252041531", "date": "2019-01-01", "price": 200, "promoted": true, "rating": 1, "type": "hat"}

```

But the \_id was created automatacllay. Result is:

```auto
{
    "took": 13,
    "errors": false,
    "items": [
        {
            "index": {
                "_index": "kr_product",
                "_type": "_doc",
                "_id": "Olcmf3IBGdIxFphTkg1Y",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 5,
                "_primary_term": 1,
                "status": 201
            }
        }
    ]
}

```

Do you have any other ideas for me? It is only the Bulk import what makes problems, but bulk import is very important for me.

---

<div class="post-metadata">

**Author:** ![psramkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/psramkumar/32/31393_2.png) [@psramkumar](https://discuss.elastic.co/u/psramkumar)\
**Post date:** [June 7, 2020, 3:27pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/8 "2020-06-07T15:27:55Z")

</div>

type. is removed in 7.7v [https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html)

---

<div class="post-metadata">

**Author:** ![elasticitm](https://avatars.discourse-cdn.com/v4/letter/e/9fc29f/32.png) [@elasticitm](https://discuss.elastic.co/u/elasticitm)\
**Post date:** [June 7, 2020, 10:07pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/9 "2020-06-07T22:07:28Z")

</div>

Oh yes I see.

Does anyone has an idea how to fix the bulk-import changeing the "\_id" issue? 😓

---

<div class="post-metadata">

**Author:** ![Bernt\_Rostad](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)\
**Post date:** [June 8, 2020, 7:02am UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/10 "2020-06-08T07:02:40Z")

</div>

> [@elasticitm](#):
>
> Does anyone has an idea how to fix the bulk-import changeing the "\_id" issue?

I maintain a system where I generate the document IDs before performing bulk index, update and delete operations using these IDs. The main difference from your example is that I use **"id"** and **"index"** , without a leading underscore, rather than "\_id" and "\_index".

Here's how I build up my bulk operation (in the Perl programming language):

```
   for (my $doc (@doc_array)) {
        my $payload = { # build a new payload
            id => $doc->{id},
            type => $fixed_type, # TODO: remove in ES7
            index => $indexname,
        };

        if ($doc->{action} ne 'delete' ) { # add document as 'source' or 'doc' depending on the action
            if ($doc->{action} eq 'update' ) { # just add the partial 'doc'
                $payload->{doc} = $doc->{partial};
                $payload->{doc_as_upsert} = 'false';
                $payload->{detect_noop} = 'true';
            } else { # for new documents add the full document in 'source'
                $payload->{source} = $doc->{full};
                $payload->{pipeline} = $pipeline if $pipeline; # only supported for indexing new docs
            }
        }
        $bulk->add_action( $doc->{action} => $payload );
   }

```

I hope this solves your problem. Good luck!

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [June 8, 2020, 6:26pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/11 "2020-06-08T18:26:46Z")

</div>

Maybe you should check the curly brace positions.

> [@elasticitm](#):
>
> `{"index":{"_index":"{{domain}}_product"}, "_id": "709_dis__29618840141927_252041531",}`

How about

```auto
{"index":{"_index":"{{domain}}_product", "_id": "709_dis__29618840141927_252041531"}}

```

---

<div class="post-metadata">

**Author:** ![elasticitm](https://avatars.discourse-cdn.com/v4/letter/e/9fc29f/32.png) [@elasticitm](https://discuss.elastic.co/u/elasticitm)\
**Post date:** [June 12, 2020, 9:35pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/12 "2020-06-12T21:35:52Z")

</div>

Hi Jörg,

That was the problem - works perfectly now! 🙂

Thanks so much for your help! 👌

Best Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 10, 2020, 9:36pm UTC](https://discuss.elastic.co/t/need-own-unqiue-id-with-bulk-insert/235599/13 "2020-07-10T21:36:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.