Getting error while restoring snapshots using curator

Elastic Stack Elasticsearch
1

Hi Team,

I am taking snapshots using curator through GCS bucket and trying to restore it . But, I am getting error as below:
2020-10-13 06:08:35,293 INFO Preparing Action ID: 1, "restore"
2020-10-13 06:08:35,293 INFO Creating client object and testing connection
2020-10-13 06:08:35,297 INFO Instantiating client object
2020-10-13 06:08:35,297 INFO Testing client connectivity
2020-10-13 06:08:35,306 INFO Successfully created Elasticsearch client object with provided settings
2020-10-13 06:08:35,308 INFO Trying Action ID: 1, "restore": taking snapshots
2020-10-13 06:08:38,111 INFO Restoring indices "['.kibana_task_manager_1', 'elastic-curator', 'instagram', 'google', 'whatsapp', '.kibana_1', '.security-7', 'facebook', '.apm-agent-configuration']" from snapshot: curator-20201013060714
2020-10-13 06:08:39,122 ERROR Failed to complete action: restore. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[curator-restrore:curator-20201013060714/oqCiYeddRAe923rE6vTHVg] cannot restore index [whatsapp] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')

Below is my curator-config file:

actions:

    1:

      action: restore

      description: taking snapshots 

      options:

        repository: curator-restrore

        name: 

        indices:

        rename_pattern: 'index(.+)'

        rename_replacement: 'restored_index$1'

      filters:

      - filtertype: age

        source: creation_date

        direction: older

        unit: minutes

        unit_count: 1

Please help me , if have done wrong while configuring. Every time, I have to close the system indices through API s. Then only restore is working fine. How to avoid that. Even though I am using replacement fields in the config file, they are not working. please suggest me with correct curator config

Please anybody reply to this question soon. Thanks in advance.

2

Actually, I want to do a scheduled restore .I can able to restore only one time. For the second job, it was throwing error as below:
2020-10-13 06:08:39,122 ERROR Failed to complete action: restore. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[curator-restrore:curator-20201013060714/oqCiYeddRAe923rE6vTHVg] cannot restore index [whatsapp] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')

Please anyone suggest me on this

3

What is the use case behind a scheduled restore?

4

Thanks for replying back. I want this for Diasater recovery. I am using curator to take snapshot from active cluster and restore snapshot to passive cluster. I am able to do scheduled backups using cron job. But not able to do scheduled restores to passive cluster. Only one job is running successful. From the 2nd time, it is throwing error as above. Please suggest me how can I achieve that.

5

OK, so you have a central snapshot repository and are writing to it from one cluster and reading from another. When you restore a snapshot you restore the full state so indices you restore can not already exist. To make this work you would therefore probably need to delete the indices from the destination cluster before restoring. Am not sure if curator can do that. This would always restore all data unlike when you take a snapshot and only altered segments are copied.

6

Do you have any idea of how to achieve it by using options.
I tried by using the option "ignore_unavailable: True". But still not working.

Is it possible to take scheduled restores or not?

7

I have not seen it used this way, but that does not necessarily mean it is not possible. As you need to delete indices before restoring and you probably do not want to re-restore all indices it may be beyond what Curator can handle out of the box and you may need to script this yourself. Others with more Curator experience may have other ideas though.

8

Yeah . Thank you @Christian_Dahlqvist. I am also thinking to use delete indices before restoring. Thanks for the help.

9

I have done backup of 13.2 tb data using curator and trying to restore the entire data to another cluster using curator. I am closing the all indices before restoring the data . Even though I am getting error as below:

<class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[diaster:curator-20201023000010/ln0mqdlgStmXa-H_s9Innw] cannot restore index [filebeat-2020.10.19] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')

I am configuring curator as below:

actions:

    1:

      action: close

      description: "close selected indices"

      options:

        ignore_empty_list: True

        continue_if_exception: True

      filters:

      - filtertype: age

        source: creation_date

        direction: older

        unit: minutes

        unit_count: 1

    2:

      action: restore

      description: restoring snapshots 

      options:

        repository: diaster

        name: 

        indices:

        ignore_empty_list: True

      filters:

      - filtertype: state

        state: SUCCESS

Anyone has any idea on it how to to get rid of that error.

10

I do not know as I have never used Curator for restoration of snapshots.

11

Anyone in your team can help me about this

12

I am able to get rid of this error when I scheduled the restore job to every one hour. Did any one has any idea. Please help me on this..I also saw some indices are missing while restoring...

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.