Restore from a 3-Node cluster to a stand alone cluster

Elastic Stack Elasticsearch
1

I have a test environment with a 3-node cluster and a stand alone cluster.

I created an index ('products') in the 3-node cluster and then performed a snapshot. I deleted the index and then use restore API to restore it back. It was successful so the the snapshot has no problem.

I copied the snapshot directory to the stand alone cluster. I registered the directory location (/home/snapshot_2) and the instance seems to recognize the snapshot properly.

[root@localhost]# curl -X GET "localhost:9201/_snapshot/backup/_all?pretty"
{
  "snapshots" : [
    {
      "snapshot" : "snapshot_1",
      "uuid" : "BNVDt0f0TqO-Ep4uQaUVkQ",
      "repository" : "backup",
      "version_id" : 7140199,
      "version" : "7.14.1",
      "indices" : [
        "my-index-000001",
        ".geoip_databases"
      ],
      "data_streams" : [ ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2021-09-10T07:48:29.217Z",
      "start_time_in_millis" : 1631260109217,
      "end_time" : "2021-09-10T07:48:52.869Z",
      "end_time_in_millis" : 1631260132869,
      "duration_in_millis" : 23652,
      "failures" : [ ],
      "shards" : {
        "total" : 4,
        "failed" : 0,
        "successful" : 4
      },
      "feature_states" : [
        {
          "feature_name" : "geoip",
          "indices" : [
            ".geoip_databases"
          ]
        }
      ]
    },
    {
      "snapshot" : "snapshot_2",
      "uuid" : "sRnJwSBUQu6PmyBMhTU_JQ",
      "repository" : "backup",
      "version_id" : 7140199,
      "version" : "7.14.1",
      "indices" : [
        "products",
        ".geoip_databases"
      ],
      "data_streams" : [ ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2021-09-16T06:22:25.740Z",
      "start_time_in_millis" : 1631773345740,
      "end_time" : "2021-09-16T06:22:51.707Z",
      "end_time_in_millis" : 1631773371707,
      "duration_in_millis" : 25967,
      "failures" : [ ],
      "shards" : {
        "total" : 2,
        "failed" : 0,
        "successful" : 2
      },
      "feature_states" : [
        {
          "feature_name" : "geoip",
          "indices" : [
            ".geoip_databases"
          ]
        }
      ]
    }
  ]
}

But when I tried to restore the 'products' index from 'snapshot_2' I'm getting the following errors:

          "explanation" : "shard has failed to be restored from the snapshot [backup:snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ] because of [failed shard on node [4V7DyqL6QOmAicP7ydZcOg]: failed recovery, failure RecoveryFailedException[[products][0]: Recovery failed on {node-1}{4V7DyqL6QOmAicP7ydZcOg}{L6MTJXFDTJqrW24fyRejgg}{127.0.0.1}{127.0.0.1:9301}{cdfhilmrstw}{ml.machine_memory=3973398528, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1988100096}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ]]; nested: SnapshotMissingException[[backup:snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ] is missing]; nested: 
NoSuchFileException[/home/snapshot2/indices/zofyJxIzTu2RAaWAcToh2w/0/snap-sRnJwSBUQu6PmyBMhTU_JQ.dat]; ] - manually close or delete the index [products] in order to retry to restore the snapshot again or use the reroute API to force the allocation of an empty primary shard"

From the error it seems it is looking for a file in a particular location

/home/snapshot2/indices/zofyJxIzTu2RAaWAcToh2w/0/snap-sRnJwSBUQu6PmyBMhTU_JQ.dat

This file exists but not in this location but on the root directory.

drwxr-xr-x. 4 elasticsearch elasticsearch  248 Sep 21 11:42 .
-rw-r--r--. 1 elasticsearch elasticsearch    8 Sep 16 14:22 index.latest
-rw-r--r--. 1 elasticsearch elasticsearch 1399 Sep 16 14:22 index-1
-rw-r--r--. 1 elasticsearch elasticsearch  359 Sep 16 14:22 snap-sRnJwSBUQu6PmyBMhTU_JQ.dat
drwxr-xr-x. 5 elasticsearch elasticsearch   96 Sep 16 14:22 indices
-rw-r--r--. 1 elasticsearch elasticsearch 9370 Sep 16 14:22 meta-sRnJwSBUQu6PmyBMhTU_JQ.dat
drwxr-xr-x. 5 root          root            57 Sep 10 15:49 ..
-rw-r--r--. 1 elasticsearch elasticsearch 9374 Sep 10 15:48 meta-BNVDt0f0TqO-Ep4uQaUVkQ.dat
-rw-r--r--. 1 elasticsearch elasticsearch  361 Sep 10 15:48 snap-BNVDt0f0TqO-Ep4uQaUVkQ.dat
drwxr-xr-x. 2 elasticsearch elasticsearch  102 Sep 10 15:46 tests-HaarN69LQ1CnQ9f641bVvQ

Anyone knows what I am doing wrong here?

Thank you.

2

Welcome to our community! :smiley:

Did you happen to try any of these options?

3

Hi,

Yes, I did

[root@localhost snapshot2]# curl -X DELETE "localhost:9201/products"
{"acknowledged":true}[root@localhost snapshot2]# 
[root@localhost snapshot2]# curl -X POST "localhost:9201/_snapshot/backup/snapshot_2/_restore?pretty" -H 'Content-Type: application/json' -d'
> {
>   "indices": "products",
>   "index_settings": {
>     "index.number_of_replicas": 0
>   }
> }
> '
{
  "accepted" : true
}
[root@localhost snapshot2]# curl -X GET "localhost:9201/_cluster/allocation/explain?pretty"
{
  "index" : "products",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2021-09-21T06:13:39.065Z",
    "failed_allocation_attempts" : 5,
    "details" : "failed shard on node [4V7DyqL6QOmAicP7ydZcOg]: failed recovery, failure RecoveryFailedException[[products][0]: Recovery failed on {node-1}{4V7DyqL6QOmAicP7ydZcOg}{L6MTJXFDTJqrW24fyRejgg}{127.0.0.1}{127.0.0.1:9301}{cdfhilmrstw}{ml.machine_memory=3973398528, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1988100096}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ]]; nested: SnapshotMissingException[[backup:snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ] is missing]; nested: NoSuchFileException[/home/snapshot2/indices/zofyJxIzTu2RAaWAcToh2w/0/snap-sRnJwSBUQu6PmyBMhTU_JQ.dat]; ",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "4V7DyqL6QOmAicP7ydZcOg",
      "node_name" : "node-1",
      "transport_address" : "127.0.0.1:9301",
      "node_attributes" : {
        "ml.machine_memory" : "3973398528",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "1988100096"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "restore_in_progress",
          "decision" : "NO",
          "explanation" : "shard has failed to be restored from the snapshot [backup:snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ] because of [failed shard on node [4V7DyqL6QOmAicP7ydZcOg]: failed recovery, failure RecoveryFailedException[[products][0]: Recovery failed on {node-1}{4V7DyqL6QOmAicP7ydZcOg}{L6MTJXFDTJqrW24fyRejgg}{127.0.0.1}{127.0.0.1:9301}{cdfhilmrstw}{ml.machine_memory=3973398528, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=1988100096}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ]]; nested: SnapshotMissingException[[backup:snapshot_2/sRnJwSBUQu6PmyBMhTU_JQ] is missing]; nested: NoSuchFileException[/home/snapshot2/indices/zofyJxIzTu2RAaWAcToh2w/0/snap-sRnJwSBUQu6PmyBMhTU_JQ.dat]; ] - manually close or delete the index [products] in order to retry to restore the snapshot again or use the reroute API to force the allocation of an empty primary shard"
        }
      ]
    }
  ]
}
1 Like
4

There should be a file called snap-sRnJwSBUQu6PmyBMhTU_JQ.dat in both locations, although it's not the same file.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.