# How to set “\_id” value in elasticsearch document as my custom document id **URL:** \ **Category:** Elasticsearch\ **Created:** [November 10, 2020, 4:00pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944 "2020-11-10T16:00:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![smruthip](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Post date:** [November 10, 2020, 4:00pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/1 "2020-11-10T16:00:28Z")
Can I do this outside in my python script before I send the data to elastic search? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 10, 2020, 4:25pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/2 "2020-11-10T16:25:47Z")
Something like: ```auto PUT test/_doc/1234 { "id": "1234" } ``` ---
**Author:** ![smruthip](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Post date:** [November 10, 2020, 4:40pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/3 "2020-11-10T16:40:28Z")
@dadoonet - thank you. I actually want to set it to a field in my document. example : currDetails\_container\_id which is in my json document , I want to set that as \_id so that it removed duplicates and just updates this id as each document with same ID comes ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 10, 2020, 4:46pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/4 "2020-11-10T16:46:56Z")
If you can't set it yourself on the client side, you can use an ingest pipeline like: ```auto POST _ingest/pipeline/_simulate { "pipeline": { "processors": [ { "set": { "field": "_id", "value": "{{currDetails_container_id}}" } } ] }, "docs": [ { "_source": { "currDetails_container_id": "1234" } } ] } ``` This gives: ```auto { "docs" : [ { "doc" : { "_index" : "_index", "_type" : "_doc", "_id" : "1234", "_source" : { "currDetails_container_id" : "1234" }, "_ingest" : { "timestamp" : "2020-11-10T16:46:08.519079145Z" } } } ] } ``` ---
**Author:** ![smruthip](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Post date:** [November 10, 2020, 6:32pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/5 "2020-11-10T18:32:15Z")
@dadoonet - thank you very much. I want this applied to all incoming documents , so I dont want to specify document in the post. I tried just POST \_ingest/pipeline/\_simulate { "pipeline": { "processors": [ { "set": { "field": "\_id", "value": "{{currDetails\_container\_id}}" } } ] } } and this gave an error that [docs] required property is missing. I have real time streaming data in my scenario and want the pipeline to update as the data is coming in can I do something like this? PUT \_ingest/pipeline/my\_pipeline { "description": "updates \_id with container id at the time of ingestion", "processors": [ { "set": { "field": "\_id", "value": "{{currDetails\_container\_id}}" } } ] } ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 10, 2020, 9:32pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/6 "2020-11-10T21:32:03Z")
Sure. Read the documentation about [ingest pipelines](https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html). Then feel free to ask about what you don't understand. I'll be happy to help. ---
**Author:** ![smruthip](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Post date:** [November 11, 2020, 1:29am UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/8 "2020-11-11T01:29:12Z")
Hi @dadoonet, Thank you very much for helping me with this. I am able to update the id with the container\_id, however its not giving me the latest document, some documents are not getting updated. Do you know how I can ensure that the document is getting updated with the latest document? there is a timefield in my document and I can see that the last document being shown is an old document and it never got updated. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 11, 2020, 5:30am UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/9 "2020-11-11T05:30:35Z")
I have no idea of what you have and what you exactly done so I can not help. Could you provide a full recreation script as described in [About the Elasticsearch category](https://discuss.elastic.co/t/about-the-elasticsearch-category/21). It will help to better understand what you are doing. Please, try to keep the example as simple as possible. A full reproduction script is something anyone can copy and paste in Kibana dev console, click on the run button to reproduce your use case. It will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer. ---
**Author:** ![smruthip](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@smruthip](https://discuss.elastic.co/u/smruthip)\ **Post date:** [November 12, 2020, 6:35pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/10 "2020-11-12T18:35:15Z")
I am so sorry. My bad! I will be more clear on my requests next time. The solution you recommended worked perfectly for me. Thank you! For others who are having the same issue: This is the query I ran. With this, my \_id is being replaced by container\_id thus removing all duplicates and updating with the latest container state ```auto PUT my_index_2020-11-11 { "settings": { "default_pipeline": "dailyindex" } } PUT _ingest/pipeline/dailyindex { "description": "updates _id with container id at the time of ingestion", "processors": [ { "set": { "field": "_id", "value": "{{container_id}}" } } ] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 10, 2020, 6:35pm UTC](https://discuss.elastic.co/t/how-to-set-id-value-in-elasticsearch-document-as-my-custom-document-id/254944/11 "2020-12-10T18:35:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.