Need own unqiue ID with Bulk insert

Elastic Stack Elasticsearch
1

Hallo @all,

I've switched from a older elasticsearch version to the current 7.7 and now getting troubles using unique id's.

In the newer elasticsearch version the "_id" is set automatically to a short one and this breaks the whole logic of my implementation.

My product id (uuid) is for example: 709_dis__29618840141927_252041531

if I import products in bulk and the id exits the product should be overwritte for example:

PUT {{base_url}}/_bulk
{"index":{"_index":"{{domain}}_product"}, "_id": "709_dis__29618840141927_252041531",}
{"date": "2019-01-01", "price": 200, "promoted": true, "rating": 1, "type": "hat"}

I know:

  • "better performance with shorter id" - it is not an option!
  • "using hash to shorten" - it is not an option!

Is there a work arrount for setting own unique id with my need length (if ES is slower in this case thats finde for me)?

Thank you for your help!

2

@elasticitm Hey, maybe the fingerprint filter / processor could be an option for you? Check this article to get an idea => https://www.elastic.co/blog/logstash-lessons-handling-duplicates

Grtz

3

Hi Willem,

thanks for your replay! Properly it could solve the problem, but makes the system more compley, so I would like to avoid using locklash.

I found out, that it works very fine when I add a single product instead of bulk:

PUT {{base_url}}/{{domain}}_product/_doc/709_dis__29618840141927_252041531

Question is now if there is a working syntax for bulk insert updates, that allows to set with own "_id"?

Best Regards

4

Can you try setting _type to _doc in the bulk request and see if that changes anything?

5

Hi Christian,

thanks for this hint :call_me_hand:

setting _doc like this

PUT {{base_url}}/_doc/_bulk

brings no effect

I'm not sure how I can set type here. Could you help me with the code?

Best Regards

6

I meant putting it in each bulk header next to the _id.

7

Hi, Christian,

unforunetly that doesn't fix the problem.

Here is my request

PUT {{base_url}}/_bulk
{"index":{"_index":"{{domain}}_product"}, "_id": "709_dis__29618840141927_252041531", "_type":"_doc"}
{"id": "709_dis__29618840141927_252041531", "date": "2019-01-01", "price": 200, "promoted": true, "rating": 1, "type": "hat"}

But the _id was created automatacllay. Result is:

{
    "took": 13,
    "errors": false,
    "items": [
        {
            "index": {
                "_index": "kr_product",
                "_type": "_doc",
                "_id": "Olcmf3IBGdIxFphTkg1Y",
                "_version": 1,
                "result": "created",
                "_shards": {
                    "total": 2,
                    "successful": 1,
                    "failed": 0
                },
                "_seq_no": 5,
                "_primary_term": 1,
                "status": 201
            }
        }
    ]
}

Do you have any other ideas for me? It is only the Bulk import what makes problems, but bulk import is very important for me.

8

type. is removed in 7.7v https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

9

Oh yes I see.

Does anyone has an idea how to fix the bulk-import changeing the "_id" issue? :sweat:

10

I maintain a system where I generate the document IDs before performing bulk index, update and delete operations using these IDs. The main difference from your example is that I use "id" and "index", without a leading underscore, rather than "_id" and "_index".

Here's how I build up my bulk operation (in the Perl programming language):

   for (my $doc (@doc_array)) {
        my $payload = { # build a new payload
            id    => $doc->{id},
            type  => $fixed_type, # TODO: remove in ES7
            index => $indexname,
        };

        if ($doc->{action} ne 'delete' ) { # add document as 'source' or 'doc' depending on the action
            if ($doc->{action} eq 'update' ) { # just add the partial 'doc'
                $payload->{doc}           = $doc->{partial};
                $payload->{doc_as_upsert} = 'false';
                $payload->{detect_noop}   = 'true';
            } else { # for new documents add the full document in 'source'
                $payload->{source}        = $doc->{full};
                $payload->{pipeline}      = $pipeline if $pipeline; # only supported for indexing new docs
            }
        }
        $bulk->add_action( $doc->{action} => $payload );
   }

I hope this solves your problem. Good luck!

11

Maybe you should check the curly brace positions.

How about

{"index":{"_index":"{{domain}}_product", "_id": "709_dis__29618840141927_252041531"}}
12

Hi Jörg,

That was the problem - works perfectly now! :slight_smile:

Thanks so much for your help! :ok_hand:

Best Regards

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.