Curator Restore action failure

Elastic Stack Elasticsearch
1

Hi,

I have been using the Curator for restoring an entire cluster's data to a backup cluster.
Below is the action file:

actions:
  1:
    action: close
    description: "Close indices before restoring snapshot"
    options:
      continue_if_exception: True
      ignore_empty_list: True
    filters:
      - filtertype: pattern
        kind: regex
        value: ".*$"
  2:
    action: restore
    description: "Restore all indices in the most recent snapshot with state SUCCESS"
    options:
      repository: "repository_name"
      name:
      indices:
	  skip_repo_fs_check: True
      wait_for_completion: True
    filters:
      - filtertype: state
        state: SUCCESS
  3:
    action: open
    description: "Open indices after restoring snapshot"
    filters:
      - filtertype: pattern
        kind: regex
        value: ".*$"

I have been able to restore the cluster twice (using different snapshot of different timings), the passwords are reset and everything is changed accordingly. I re-run the curator but with passwords that have been restored via the snapshot but the third time I try to restore using the latest snapshot, I face an error : Authentication failure. I am no longer able login using the elastic credentials.

2018-12-10 12:40:54,874 DEBUG elasticsearch log_request_success:85 > None
2018-12-10 12:40:54,874 DEBUG elasticsearch log_request_success:86 < {"acknowledged":true}
2018-12-10 12:40:54,875 INFO curator.cli run:196 Action ID: 1, "close" completed.
2018-12-10 12:40:54,875 DEBUG curator.cli run:126 action_disabled = False
2018-12-10 12:40:54,875 DEBUG curator.cli run:130 continue_if_exception = False
2018-12-10 12:40:54,875 DEBUG curator.cli run:132 timeout_override = 21600
2018-12-10 12:40:54,875 DEBUG curator.cli run:134 ignore_empty_list = False
2018-12-10 12:40:54,875 DEBUG curator.cli run:136 allow_ilm_indices = False
2018-12-10 12:40:54,875 INFO curator.cli run:146 Preparing Action ID: 2, "restore"
2018-12-10 12:40:54,876 DEBUG curator.utils get_client:803 kwargs = {'hosts': ['<>'], 'port': 9200, 'use_ssl': True, 'certificate': '.crt', 'http_auth': 'elastic:abced', 'master_only': False, 'aws_token': None, 'aws_key': None, 'aws_sign_request': False, 'ssl_no_validate': False, 'client_key': None, 'aws_secret_key': None, 'client_cert': None, 'url_prefix': '', 'timeout': 21600}
2018-12-10 12:40:54,876 DEBUG curator.utils get_client:809 Attempting to verify SSL certificate.
2018-12-10 12:40:54,886 DEBUG curator.utils get_client:880 Not using "requests_aws4auth" python module to connect.
2018-12-10 12:40:54,887 DEBUG urllib3.util.retry from_int:200 Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2018-12-10 12:40:54,887 DEBUG urllib3.connectionpool _new_conn:824 Starting new HTTPS connection (1):
2018-12-10 12:40:54,915 DEBUG urllib3.connectionpool _make_request:396 https://<>:9200 "GET / HTTP/1.1" 401 345
2018-12-10 12:40:54,916 WARNING elasticsearch log_request_fail:97 GET https://<>:9200/ [status:401 request:0.029s]
2018-12-10 12:40:54,916 DEBUG elasticsearch log_request_fail:105 > None
2018-12-10 12:40:54,916 DEBUG elasticsearch log_request_fail:110 < {"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

2

Curator has no control over authentication. It just passes the response through from Elasticsearch. If it can’t connect, that’s at the Elasticsearch level. This is not something Curator can address or fix.

3

I see, Thank you.

This is the first time I am using the Curator tool for restore, I have used it for snapshot before and that was quite an easy process. But now, I wanted to use it in the scenario that I need to restore the entire data of the main cluster to a backup cluster.
So I wanted to understand how the Restore works via Curator tool. Is there anything I need to verify or execute manually after the tool has run the restore action? Other than the usual health check of the cluster and verifying if I can login with the main cluster's elastic credentials after restore.
I know that the tool runs various checks while executing the restore action but I wanted to understand a bit in depth regarding which aspects the Curator tool has control over while executing the restore action.

4

Curator really only does the same things the Restore API does with Elasticsearch. If it completes without error, Curator considers it done.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.