Restore a snapshot error ULTRAWARM INTERNAL EXCEPTION

Elastic Stack Elasticsearch
1

I'm trying to restore an index with curator on an Amazon Elasticsearch domain (version 6.8), but the operation seems to failed with the error ULTRAWARM INTERNAL EXCEPTION.
The curator restore yaml is:

    actions:
   1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository: my-repo
      name:
      indices:
        - my_index
      extra_settings:
       index_settings:
        number_of_replicas: 0
        index.auto_expand_replicas: true
       ignore_index_settings:
       - index.refresh_interval
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: pattern
      kind: prefix
      value: curator-
    - filtertype: state
      state: SUCCESS

The curator output is:

    2020-04-10 11:24:29,779 INFO      Preparing Action ID: 1, "restore"
    2020-04-10 11:24:29,779 INFO      Creating client object and testing connection
    2020-04-10 11:24:29,871 INFO      Credentials found in config file: ~/.aws/config
    2020-04-10 11:24:29,874 INFO      Instantiating client object
    2020-04-10 11:24:29,874 INFO      Testing client connectivity
    2020-04-10 11:24:30,223 INFO      Successfully created Elasticsearch client object with provided 
    settings
    2020-04-10 11:24:30,278 INFO      Trying Action ID: 1, "restore": Restore all indices in the most 
    recent snapshot with state SUCCESS.  Wait for the restore to complete before continuing.  Do 
    not skip the repository filesystem access check.  Use the other options to define the 
    index/shard 
    settings for the restore.
    2020-04-10 11:24:31,017 INFO      Restoring indices "['my_index']" from snapshot: 
    curator-20200410072900
    2020-04-10 11:24:31,239 INFO      Provided indices: ['my_index']
    2020-04-10 11:24:31,239 INFO      Found indices: ['my_index']
    2020-04-10 11:24:31,239 INFO      Index "my_index" is still in stage "INDEX"
    2020-04-10 11:24:41,299 INFO      _recovery returned an empty response. Trying again.
    2020-04-10 11:24:51,359 INFO      _recovery returned an empty response. Trying again.
    2020-04-10 11:25:01,422 INFO      _recovery returned an empty response. Trying again.

This goes on indefinitely.

When I call

    POST  _cluster/allocation/explain?pretty

I get the following error:

    {
      "index": "my_index",
      "shard": 4,
    "primary": true,
    "current_state": "unassigned",
    "unassigned_info": {
      "reason": "ALLOCATION_FAILED",
      "at": "2020-04-10T09:24:34.518Z",
      "failed_allocation_attempts": 1,
      "details": "[ULTRAWARM INTERNAL EXCEPTION]",
      "last_allocation_status": "no"
     },
     "can_allocate": "no",
     "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
    "node_allocation_decisions": [
      {
        "node_id": "xhpwVdgrR9ia7ly_jXAYhA",
        "node_name": "xhpwVdg",
        "node_decision": "no",
        "weight_ranking": 1,
        "deciders": [
          {
            "decider": "restore_in_progress",
            "decision": "NO",
            "explanation": "[ULTRAWARM INTERNAL EXCEPTION]"
          }
        ]
      }
    ]
}

Both the source and destination domains aren't using the ULTRAWARM feature of AWS.
Is there anything I need to consider?

Thanks!

1 Like
2

You will likely need to ask this question of your AWS support contacts, since this message is coming from some of AWS's unsupported customisations of Elasticsearch that we can't see or help debug. We don't often see AWS folk around here, sorry.

3

Hi David,
thanks anyway for your super fast reply.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.