How to create an index in ES6 that automatically assigns a Field value (Ex:- employee_Id) found inside a document to _id (Document ID) each time a new document is inserted

How to create an index in ES6 that automatically assigns a Field value (Ex:- employee_Id) found inside a document to _id (Document ID) each time a new document is inserted

You can create an ingest pipeline that you can call when you index a document. It can set the _id from another field. See: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/script-processor.html

Thanks for sharing the above link, that helped me understand.

Although I was left with one more question based on the below scenario what I have:

  • I am trying to push JSON document to Elastic using Fluentd.
  • Fluentd connects to Elastic and Push document to a index
  • As per the requirement we need to do a POST call with Unique ID [employee_Id] each time to insert new document or re-insert the whole document with new data (If document ID already exist).
    Ex:- Below 12345 - is the employee_Id
    post Employeedata/Employees/12345
    {
    "employee_id": "12345",
    "title": "Programmer",
    "Desc": "Java Dev",
    "Sal":"9100.00"
    }
  • So if Elastic receives the data from Fluentd to write to a index, is there any way to make Elastic index intelligent to automaticcaly identify the employee_id from the document, make it as _id and insert the record.

If you are on Elasticsearch 6.5 or higher you can define a default pipeline for the index.

1 Like

Great. I didn't know that this was merged. :wink:

Thanks Christian, for letting me know this possibility. This would help us in solving a long standing issue. Can you please share a code snippet (or) example (or) steps on how "index.default_pipeline" can be included while index creation.

I appreciate your help in advance.

Hi,

Here how to add a default_pipline settings on an index.

PUT /YOUR_INDEX/_settings
{
    "index" : {
        "default_pipeline" : "YOUR_PIPELINE_NAME"
    }
}

doc is here: https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-update-settings.html

As it's a dynamic setting it will apply without any other manipulation.

dynamic

They can be changed on a live index using the update-index-settings API.

If you are using time-based indices you can also apply it via an index template.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.