# What's the recommended strategy to reindex all (over 500) indices from 5.6.3 to 7.x?

**URL:** <https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124>\
**Category:** Elasticsearch\
**Created:** [July 12, 2019, 12:02am UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124 "2019-07-12T00:02:37Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![shradhatx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shradhatx/32/45398_2.png) [@shradhatx](https://discuss.elastic.co/u/shradhatx)\
**Post date:** [July 12, 2019, 12:02am UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/1 "2019-07-12T00:02:37Z")

</div>

I have to migrate from ES5.6.3 to latest ES 7.x There are over 500 indices, 10 flavor of index, high number of index because of rollover ex index1-yyyy.mm and so on. The two options can be -

1. scripting - ex bash script connecting to new cluster  
read all indexes from a file (this is done separately by connecting to 5.x cluster)  
loop  
create mapping  
reindex from remote

Question- can index of same flavor be reindexed api in a single command with source and dest index as "index1-\*"

1. same logic in a java program using high level REST client

Since the available servers to work on is Windows, I am having difficulty setting up bash script, If there is a way to do it in one shot for indexes of same flavor I would like that since it is a one time job.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 12, 2019, 6:22am UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/2 "2019-07-12T06:22:14Z")

</div>

Not sure I fully understand but did you look at the reindex API?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 7:20am UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/3 "2019-07-12T07:20:43Z")

</div>

[Reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/docs-reindex.html#reindex-from-remote) supports reindexing from a remote Elasticsearch cluster:

```
POST _reindex
{
"source": {
"remote": {
  "host": "http://otherhost:9200",
  "username": "user",
  "password": "pass"
},
"index": "source",
"query": {
  "match": {
    "test": "data"
  }
}
},
"dest": {
"index": "dest"
}
}
```

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 7:36am UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/4 "2019-07-12T07:36:16Z")

</div>

I guess also you can Reindex multiple indices with a wildcard, check this link [https://www.elastic.co/guide/en/elasticsearch/reference/7.2/docs-reindex.html#\_reindex\_daily\_indices](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/docs-reindex.html#_reindex_daily_indices)

You can also try [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/reindex.html)

---

<div class="post-metadata">

**Author:** ![shradhatx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shradhatx/32/45398_2.png) [@shradhatx](https://discuss.elastic.co/u/shradhatx)\
**Post date:** [July 12, 2019, 4:53pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/5 "2019-07-12T16:53:09Z")

</div>

Thanks @ylasri and @dadoonet . I tried reindex using wildcard with painless script. I have many indices Ex: audit-2018.01, audit-2018.02 and so on and want to create in another instance the indices by the same name. What's the correct way to reference remote index in the script?  
POST \_reindex  
{  
"conflicts": "proceed",  
"source": {  
"remote": {  
"host": "[http://localhost:9200](http://localhost:9200)"  
},  
"index": "audit\*",  
"size": 1000  
},  
"dest": {  
"index": "audit",  
"op\_type": "create"  
},  
"script": {  
"lang": "painless",  
"source": "ctx.\_index = 'audit' + (remote.\_index.substring('audit'.length(), remote.\_index.length()))  
}  
}

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 5:34pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/7 "2019-07-12T17:34:38Z")

</div>

try this with 1 small indice

```
 POST _reindex {
    "conflicts": "proceed",
    "source": {
    "remote": {
    "host": "http://localhost:9200",
      "username": "elastic",
      "password": "changeme"
    },
    "index": "audit-*",
    "size": 1
    },
    "dest": {
    "index": "audit",
    "op_type": "create"
    },
    "script": {
    "lang": "painless",
        "source": "ctx._index = 'audit-' + (ctx._index.substring('audit-'.length(), ctx._index.length()))"
    }
    }
```

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 5:54pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/8 "2019-07-12T17:54:18Z")

</div>

add this setup in remote cluster elasticsearch.yml file and restart the cluster

```
reindex.remote.whitelist: localhost:9200
```

---

<div class="post-metadata">

**Author:** ![shradhatx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shradhatx/32/45398_2.png) [@shradhatx](https://discuss.elastic.co/u/shradhatx)\
**Post date:** [July 12, 2019, 6:31pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/9 "2019-07-12T18:31:33Z")

</div>

@ylasri thank you, it does create multiple indexes. An interesting observation though (and my issue remains)  
POST \_reindex  
{  
"conflicts": "proceed",  
"source": {  
"remote": {  
"host": "[http://localhost:9200](http://localhost:9200)"  
},  
"index": "audit-2010.\*",  
"size": 100  
},  
"dest": {  
"index": "audit-2010",  
"op\_type": "create"  
},  
"script": {  
"lang": "painless",  
"source": "ctx.\_index = 'audit-2010-' + (ctx.\_index.substring('audit-2010.'.length(), ctx.\_index.length()))"  
}  
}  
creates multiple index audit-2010-01, audit-2010-02 but the below creates just one index . (the only difference is dot instead of dash since my original index are like audit-2010.01, audit-2010.01 I want to keep the same index name in the new instance

POST \_reindex  
{  
"conflicts": "proceed",  
"source": {  
"remote": {  
"host": "[http://localhost:9200](http://localhost:9200)"  
},  
"index": "audit-2010.\*",  
"size": 100  
},  
"dest": {  
"index": "audit-2010",  
"op\_type": "create"  
},  
"script": {  
"lang": "painless",  
"source": "ctx.\_index = ' **audit-2010.**' + (ctx.\_index.substring('audit-2010.'.length(), ctx.\_index.length()))"  
}  
}

Any reason why?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 6:59pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/10 "2019-07-12T18:59:39Z")

</div>

Can you share the indice name created by the second script?

---

<div class="post-metadata">

**Author:** ![shradhatx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shradhatx/32/45398_2.png) [@shradhatx](https://discuss.elastic.co/u/shradhatx)\
**Post date:** [July 12, 2019, 7:51pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/11 "2019-07-12T19:51:23Z")

</div>

@yalsri the index created is audit-2010

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 8:12pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/12 "2019-07-12T20:12:40Z")

</div>

Try this pattern "index": "audit-2010\*" instead of "index": "audit-2010.\*" may be the issue is with the special dot 🙂

---

<div class="post-metadata">

**Author:** ![shradhatx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shradhatx/32/45398_2.png) [@shradhatx](https://discuss.elastic.co/u/shradhatx)\
**Post date:** [July 12, 2019, 8:29pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/13 "2019-07-12T20:29:45Z")

</div>

@yalsri it doesn't matter the selection by "index":"audit-2010\*" . The question is how to escape the dot in painless. Escaping with \. gives error  
"source": "ctx.\_index = 'audit-2010\.' + (ctx.\_index.substring('....

@dadoonet - David, this works in painless  
"source": "ctx.\_index = (ctx.\_index.substring(1, ctx.\_index.length()))"

and this does not  
"source": "ctx.\_index = (ctx.\_index.substring( **0** , ctx.\_index.length()))"

I do not get any syntax error but the default index name is created by reindex when using the later.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [July 12, 2019, 11:15pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/14 "2019-07-12T23:15:09Z")

</div>

painfull painless 🙂  
Hi @nik9000 we need your help here if possible

---

<div class="post-metadata">

**Author:** ![shradhatx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shradhatx/32/45398_2.png) [@shradhatx](https://discuss.elastic.co/u/shradhatx)\
**Post date:** [July 15, 2019, 4:49pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/15 "2019-07-15T16:49:08Z")

</div>

@ylasri LOL - thanks for your quick responses.  
@nik9000 your help is awaited.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2019, 4:59pm UTC](https://discuss.elastic.co/t/whats-the-recommended-strategy-to-reindex-all-over-500-indices-from-5-6-3-to-7-x/190124/16 "2019-08-12T16:59:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.