How to create auto generated ids while reindexing?

[praveengadugin1 (praveen gadugin)]

On using reindex API for reindexing 5 TB of data, to make the process faster I would like to create auto generated ids for new index as mentioned here.How do we do it??

https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_disable_swapping

[gabriel_tessier]

Hi @praveengadugin1 ,

As the name suggest auto-generated is automatic, so you just need to remove the id field that you have and reindex your data elastic will assign an auto-generated one in "_id" field.

Check about ingest to remove the id field if you have one.

You can try:

POST my_index/_doc
{"foo": "bar"}

it will return:

{
  "_index" : "my_index",
  "_type" : "_doc",
  "_id" : "X80Qu2sBMcYBL_rB5lGJ",
  "_version" : 1,
  "result" : "created", 

the auto generated id is "X80Qu2sBMcYBL_rB5lGJ"

Hope it help.

[SudevPK]

How to change _id as GUID during reindex

I have two different elastic search nodes.I need to reindex 10 indices from one node to 1 index in another node.But there is not unique id's in the 10 indices, so i need to generate _id as GUID while reindexing

I am linking my old post of same
Try

POST _reindex
{
  "source": {
    "index": ["twitter", "twitter2"]
  },
  "dest": {
    "index": "new_twitter"
  },
  "script": {
    "source": "ctx._id=null",
    "lang": "painless"
  }
}

[gabriel_tessier]

Hi,

With TB of data I guess that ingest may be faster than script but better to make your own bench and share it here it may be useful for other people.

About ingest you can use remove processor to remove the id field.
https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html