# Restoring indexes with curator

**URL:** <https://discuss.elastic.co/t/restoring-indexes-with-curator/158925>\
**Category:** Elasticsearch\
**Created:** [November 30, 2018, 3:14pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925 "2018-11-30T15:14:31Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dmytro\_Kulyk](https://avatars.discourse-cdn.com/v4/letter/d/7ba0ec/32.png) [@Dmytro\_Kulyk](https://discuss.elastic.co/u/Dmytro_Kulyk)\
**Post date:** [November 30, 2018, 3:14pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/1 "2018-11-30T15:14:31Z")

</div>

Hi, I have a problem with restoring indexes from one cluster to another.  
I'm using curator tool with actionfile:

```
> actions:
> 1:
> action: close
> options:
> delete_aliases: False
> disable_action: False
> ignore_empty_list: True
> filters:
> - filtertype: pattern
> kind: regex
> value: '(.*audit.*|.*debug.*|.*erp.*)'
>       
> 2:
> action: restore
> options:
> repository: full_backup
> name: full_backup
> indices:
> wait_interval: 10
> max_wait: 3600
> include_aliases: False
> ignore_unavailable: True
> include_global_state: True
> partial: False
> wait_for_completion: True
> skip_repo_fs_check: True
> continue_if_exception: True
> disable_action: False
> filters:
> - filtertype: state
> state: SUCCESS

```

and when it tries to restore indexes it throughs an error:

`> ERROR Failed to complete action: restore. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: Unable to obtain recovery information for specified indices. Error: NotFoundError(404, u'index_not_found_exception', u'no such index')`

when I'm trying to restore to the same env - it restores without any errors.  
Can you please help me?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 30, 2018, 3:32pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/2 "2018-11-30T15:32:20Z")

</div>

On cluster 2, have you created the repository `full_backup` which uses the exact same data store as cluster 1? Ostensibly created with the exact same API call?

What do you see on cluster 2 when you run:

```auto
GET /_snapshot

```

Does it look exactly like what you see on cluster 1 when you run the same thing?

---

<div class="post-metadata">

**Author:** ![Dmytro\_Kulyk](https://avatars.discourse-cdn.com/v4/letter/d/7ba0ec/32.png) [@Dmytro\_Kulyk](https://discuss.elastic.co/u/Dmytro_Kulyk)\
**Post date:** [November 30, 2018, 3:50pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/3 "2018-11-30T15:50:04Z")

</div>

Yes, I did repository. It is the same folder on a shared server and it is mapped to both clusters.

On cluster 2 I see:

> {  
> "full\_backup": {  
> "type": "fs",  
> "settings": {  
> "compress": "true",  
> "location": "/mnt/elk\_full\_backup"  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [November 30, 2018, 4:40pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/4 "2018-11-30T16:40:01Z")

</div>

So, what is the output of:

```auto
GET /_snapshot/_status

```

on each cluster?

---

<div class="post-metadata">

**Author:** ![Dmytro\_Kulyk](https://avatars.discourse-cdn.com/v4/letter/d/7ba0ec/32.png) [@Dmytro\_Kulyk](https://discuss.elastic.co/u/Dmytro_Kulyk)\
**Post date:** [November 30, 2018, 5:05pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/5 "2018-11-30T17:05:13Z")

</div>

on both clusters:

> `"snapshots": []`

But using ElasticHQ and checking snapshots on each cluster I can see this snapshot on both of them.

Also with doing

> `GET /_cat/snapshots/full_backup`

I have

> full\_backup SUCCESS 1543587225 15:13:45

On both clusters.

---

<div class="post-metadata">

**Author:** ![Dmytro\_Kulyk](https://avatars.discourse-cdn.com/v4/letter/d/7ba0ec/32.png) [@Dmytro\_Kulyk](https://discuss.elastic.co/u/Dmytro_Kulyk)\
**Post date:** [December 3, 2018, 10:32am UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/6 "2018-12-03T10:32:35Z")

</div>

In Debug mode I have such log:

```auto
INFO curator.utils restore_check:1616 Index "index_name" is still in stage "INDEX"
2018-12-03 11:30:18,593 DEBUG curator.utils wait_for_it:1754 Response: False
2018-12-03 11:30:18,593 DEBUG curator.utils wait_for_it:1774 Action "restore" not yet complete, 169 total seconds elapsed. Waiting 9 seconds before checking again.
2018-12-03 11:30:27,598 DEBUG curator.utils wait_for_it:1751 Elapsed time: 181 seconds
2018-12-03 11:30:27,799 ERROR curator.cli run:186 Failed to complete action: restore. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: Unable to obtain recovery information for specified indices. Error: NotFoundError(404, u'index_not_found_exception', u'no such index')

```

what can it be? can you please help me?  
Maybe you need more information? Just tell me what info you need.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [December 3, 2018, 4:30pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/7 "2018-12-03T16:30:38Z")

</div>

> [@Dmytro\_Kulyk](#):
>
> ```auto
> INFO curator.utils restore_check:1616 Index "index_name" is still in stage "INDEX"
> 
> ```

This log message, and the subsequent ones, indicate that Curator has been _trying_ to restore for 169 seconds (just under 3 minutes), and suddenly, 9 seconds later, a very unexpected log message comes. After successfully running every `9` seconds for `181` seconds (which is evident in the log messages and time stamps), suddenly, [this API call](https://github.com/elastic/curator/blob/v5.5.4/curator/utils.py#L1595) fails:

```auto
    response = client.indices.recovery(index=to_csv(index_list), human=True)

```

Curator is just trying to collect the status of recovering indices, specifically the ones in `index_list`, and one or more of them is missing, resulting in the `404` error you are seeing.

The thing is, `index_list` doesn't change over successive iterations. The `restore_check` function gets called by the `wait_for_it` function every _n_ seconds until the restore is complete. Completeness is determined by checking each index specified in `index_list` having all of its recovering shards in a `DONE` state. If a single shard in any index in `index_list` does not report `DONE`, then the `restore_check` function returns `False` to the `wait_for_it` function, which then waits _n_ seconds, and repeats the `restore_check` calls until it either returns `True`, or `max_wait` is reached. The exact same `index_list` data is sent every time. No alterations or omissions for indices already restored is performed.

In your case, the function is working flawlessly—repeatedly, even—and then stops because the index that it's trying to restore is no longer there (the `404` error). If it were a problem with the cluster, I would expect it to fail on the very first check of the recovery state. But in your case, it's clearly running just fine for around 3 minutes before it suddenly fails. Without more insight and understanding here, I'm forced to guess what's going on.

So, is there some other process that alters the indices/shards while the restore is going on? That's the first thought that comes to mind.

---

<div class="post-metadata">

**Author:** ![Dmytro\_Kulyk](https://avatars.discourse-cdn.com/v4/letter/d/7ba0ec/32.png) [@Dmytro\_Kulyk](https://discuss.elastic.co/u/Dmytro_Kulyk)\
**Post date:** [December 4, 2018, 7:38am UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/8 "2018-12-04T07:38:06Z")

</div>

No, there is no any other processes that can use indexes. As you can see in my actionfile, I'm closing all indexes before restoring them.

I've changed `include_global_state: True` to `False` and it did restore indexes, but from 52m records it did only 20m and I don't know why... Maybe I should change this parameter in backup actionfile also?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [December 4, 2018, 12:49pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/9 "2018-12-04T12:49:11Z")

</div>

All that `include_global_state` does with snapshots (and restores) is include any templates, and any persistent cluster settings. I would take a look at the snapshot state.

---

<div class="post-metadata">

**Author:** ![Dmytro\_Kulyk](https://avatars.discourse-cdn.com/v4/letter/d/7ba0ec/32.png) [@Dmytro\_Kulyk](https://discuss.elastic.co/u/Dmytro_Kulyk)\
**Post date:** [December 4, 2018, 1:17pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/10 "2018-12-04T13:17:44Z")

</div>

![Screenshot%20at%20Dec%2004%2014-16-59](https://us1.discourse-cdn.com/elastic/original/3X/d/d/dd770bfe091152c198f24a5aa601d64440021923.png)

As you can see, state of snapshot is "Success"

Also, with

> POST /\_snapshot/full\_backup/\_verify

I can see that both of clusters see that snapshot and all 6 nodes (3 in each cluster) are shows it  
Cluster1:

> {  
> "nodes": {  
> "kZjywCuhTUiEUVRjGCFMPw": {  
> "name": "elk3"  
> },  
> "Jth2U1CeTSqvmpSiBNFAYg": {  
> "name": "elk1"  
> },  
> "BbGOrWrbRT6EkERSQRtaIw": {  
> "name": "elk2"  
> }  
> }  
> }

Cluster2:

> {  
> "nodes": {  
> "6HV9p3ZTQ56\_YCYcgHu5Hg": {  
> "name": "elk1"  
> },  
> "MYXtf2LiTTOCAi-rvu21QA": {  
> "name": "elk3"  
> },  
> "qgpSoNgERN25M8sTkfD2TA": {  
> "name": "elk2"  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [December 4, 2018, 1:31pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/11 "2018-12-04T13:31:41Z")

</div>

> [@Dmytro\_Kulyk](#):
>
> but from 52m records it did only 20m

How is this count determined? What does `_cat/indices` show on each server?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2019, 1:31pm UTC](https://discuss.elastic.co/t/restoring-indexes-with-curator/158925/12 "2019-01-01T13:31:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.