# Remote reindex with wildcard to multiple indices

**URL:** <https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [November 9, 2022, 3:04pm UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552 "2022-11-09T15:04:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [November 9, 2022, 3:04pm UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/1 "2022-11-09T15:04:14Z")

</div>

Hey Everyone,

I'm having some issues with running a remote reindex from cluster a to cluster b. The end goal I'm trying to achieve is having some index pattern that the user can provide, and the remote reindex will get all the indices with that pattern from a remote cluster, and create them on the local cluster.

This should include having op\_type set to "create" as it should be used to sync the clusters in case of some failure. I tried taking the script provided by elastic here: [Reindex API | Elasticsearch Guide [8.5] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/docs-reindex.html#_reindex_daily_indices) , and just removing the part where he adds a minus (-), but that ends up failing.

This is what the script looks like currently:

```auto
POST _reindex
{
  "source": {
    "remote": {
      "host": "https://remote_host:9200",
      "username": "elastic",
      "password": "SomePass"
    },
    "index": "source-*"
  },
  "dest": {
    "index": "source",
    "op_type": "create"
  },
  "script": {
    "lang": "painless",
    "source": "ctx._index = 'source-' + (ctx._index.substring('source-'.length(), ctx._index.length()))"
  }
}

```

One more thing that needs to be said, I'm not sure if creating missing indices on the destination will break ILM. Do the reindexed indices still contain the same metadata ILM uses to tell the indices apart... their age, their order, and so on?

I'm aware some people created scripts for this in bash and so on, but I would primarily like to know if there's some way to overcome this with Elastic and it's API purely. If not, I can write the logic used in the bash scripts myself in Ansible.

Thanks in advance for any help!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 9, 2022, 10:55pm UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/2 "2022-11-09T22:55:03Z")

</div>

> [@lduvnjak](#):
>
> but that ends up failing

Failing how? It helps if you share the response from Elasticsearch, and any relevant logs.

> [@lduvnjak](#):
>
> Do the reindexed indices still contain the same metadata ILM uses to tell the indices apart... their age, their order, and so on?

Nope.

---

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [November 9, 2022, 11:10pm UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/3 "2022-11-09T23:10:59Z")

</div>

Sorry for not giving an example. There is no actual error, what ends up happening is all the data from the source indices get written to one destination index. In this example it would be "source".

Only when I make the index name different from the source do they get replicated semi-correctly. For example, by adding a minus at the end, or any other string. By semi-correctly I mean the data is correct, but of course the names of the indices are different, which is not acceptable in this situation.

Regarding the second question, is there any way to make ILM work in this situation as on the source cluster? If it's not possible in the basic license, would it be possible using CCR?

Thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 10, 2022, 6:06am UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/4 "2022-11-10T06:06:20Z")

</div>

> [@lduvnjak](#):
>
> There is no actual error, what ends up happening is all the data from the source indices get written to one destination index. In this example it would be "source".

Right, cause that's what you told it to do 🙂

> [@lduvnjak](#):
>
> By semi-correctly I mean the data is correct, but of course the names of the indices are different, which is not acceptable in this situation.

There's not currently an easy way to index multiple source indices into multiple destination ones, it's a DIY process using a for loop or something in some external code.

> [@lduvnjak](#):
>
> Regarding the second question, is there any way to make ILM work in this situation as on the source cluster?

If you reindexed into a write alias it will use the ILM policy, but it will treat the data as new and not factor in existing ILM settings on those indices.

CCR might work though.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 10, 2022, 6:18am UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/5 "2022-11-10T06:18:02Z")

</div>

Have you considered using the snapshot and restore APIs? These retain the index settings, but do copy the indices axactly as they are and do not allow you to change mappings, which you can do when reindexing.

---

<div class="post-metadata">

**Author:** ![lduvnjak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lduvnjak/32/77724_2.png) [@lduvnjak](https://discuss.elastic.co/u/lduvnjak)\
**Post date:** [November 10, 2022, 7:20am UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/6 "2022-11-10T07:20:40Z")

</div>

Yeah I figured snapshot and restore would be my next best bet. Just wanted to confirm if there's a simpler way before doing that.

The potential issue with a snapshot and restore would be the time it takes to complete. The cluster is quite large with over 2tb ingested daily. I'll see if I can make it work. Thank you for the help Mark, and Christian.

Cheers!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 10, 2022, 7:22am UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/7 "2022-11-10T07:22:25Z")

</div>

Using snapshot and restore will likely be faster and require less resources than reindexing.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2022, 7:22am UTC](https://discuss.elastic.co/t/remote-reindex-with-wildcard-to-multiple-indices/318552/8 "2022-12-08T07:22:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.