I'm trying to restore an index with curator on an Amazon Elasticsearch domain (version 6.8), but the operation seems to failed with the error ULTRAWARM INTERNAL EXCEPTION.
The curator restore yaml is:
actions:
1:
action: restore
description: >-
Restore all indices in the most recent snapshot with state SUCCESS. Wait
for the restore to complete before continuing. Do not skip the repository
filesystem access check. Use the other options to define the index/shard
settings for the restore.
options:
repository: my-repo
name:
indices:
- my_index
extra_settings:
index_settings:
number_of_replicas: 0
index.auto_expand_replicas: true
ignore_index_settings:
- index.refresh_interval
wait_for_completion: True
max_wait: 3600
wait_interval: 10
filters:
- filtertype: pattern
kind: prefix
value: curator-
- filtertype: state
state: SUCCESS
The curator output is:
2020-04-10 11:24:29,779 INFO Preparing Action ID: 1, "restore"
2020-04-10 11:24:29,779 INFO Creating client object and testing connection
2020-04-10 11:24:29,871 INFO Credentials found in config file: ~/.aws/config
2020-04-10 11:24:29,874 INFO Instantiating client object
2020-04-10 11:24:29,874 INFO Testing client connectivity
2020-04-10 11:24:30,223 INFO Successfully created Elasticsearch client object with provided
settings
2020-04-10 11:24:30,278 INFO Trying Action ID: 1, "restore": Restore all indices in the most
recent snapshot with state SUCCESS. Wait for the restore to complete before continuing. Do
not skip the repository filesystem access check. Use the other options to define the
index/shard
settings for the restore.
2020-04-10 11:24:31,017 INFO Restoring indices "['my_index']" from snapshot:
curator-20200410072900
2020-04-10 11:24:31,239 INFO Provided indices: ['my_index']
2020-04-10 11:24:31,239 INFO Found indices: ['my_index']
2020-04-10 11:24:31,239 INFO Index "my_index" is still in stage "INDEX"
2020-04-10 11:24:41,299 INFO _recovery returned an empty response. Trying again.
2020-04-10 11:24:51,359 INFO _recovery returned an empty response. Trying again.
2020-04-10 11:25:01,422 INFO _recovery returned an empty response. Trying again.
This goes on indefinitely.
When I call
POST _cluster/allocation/explain?pretty
I get the following error:
{
"index": "my_index",
"shard": 4,
"primary": true,
"current_state": "unassigned",
"unassigned_info": {
"reason": "ALLOCATION_FAILED",
"at": "2020-04-10T09:24:34.518Z",
"failed_allocation_attempts": 1,
"details": "[ULTRAWARM INTERNAL EXCEPTION]",
"last_allocation_status": "no"
},
"can_allocate": "no",
"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions": [
{
"node_id": "xhpwVdgrR9ia7ly_jXAYhA",
"node_name": "xhpwVdg",
"node_decision": "no",
"weight_ranking": 1,
"deciders": [
{
"decider": "restore_in_progress",
"decision": "NO",
"explanation": "[ULTRAWARM INTERNAL EXCEPTION]"
}
]
}
]
}
Both the source and destination domains aren't using the ULTRAWARM feature of AWS.
Is there anything I need to consider?
Thanks!