Restore index from s3

Elastic Stack Elasticsearch
1

hi i am using ES cloud managed version .
i did index backup to s3 using following command

PUT /_snapshot/hc_es_s3_repository/dtoperation_1
{

"indices": "dtoperation",

"ignore_unavailable": true,

"include_global_state": false

}

after that i deleted the index from ES
and did snapshot restore using below command -

POST /_snapshot/hc_es_s3_repository/dtoperation_1/_restore
{

"indices": "dtoperation",

"ignore_unavailable": false,

"include_global_state": false

}

both command was ran successfully and index is created , but index status is still red and document count is 0 .

any suggestion in this ?
as there is no error in above steps .

thanks

2

@DILIP_SHARMA

looks like the shards in that index failed to properly restore. Can you check the cat shards API and paste the output here so we can understand why the restore failed?

3

Hi Armin
Please find output of following command -
GET _cat/shards/dtoperation

dtoperation 1 p UNASSIGNED
dtoperation 1 r UNASSIGNED
dtoperation 0 r UNASSIGNED
dtoperation 0 p UNASSIGNED

4

@DILIP_SHARMA

we need the format from the paragraph I linked to get the reason:

GET _cat/shards?h=index,shard,prirep,state,unassigned.reason

:slight_smile:

5

dtoperation 1 p UNASSIGNED NEW_INDEX_RESTORED
dtoperation 1 r UNASSIGNED NEW_INDEX_RESTORED
dtoperation 0 r UNASSIGNED NEW_INDEX_RESTORED
dtoperation 0 p UNASSIGNED NEW_INDEX_RESTORED

6

@DILIP_SHARMA

thanks seems those shards are not getting assigned. Can you check the allocation explain API to see what's going on there?

GET /_cluster/allocation/explain
7

Please find the output

{
"index" : ".monitoring-alerts-7",
"shard" : 0,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "REPLICA_ADDED",
"at" : "2020-02-26T11:44:03.853Z",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "",
"node_name" : "",
"transport_address" : "",
"node_attributes" : {
"logical_availability_zone" : "zone-0",
"server_name" : "",
"availability_zone" : "ap-southeast-1a",
"xpack.installed" : "true",
"data" : "warm",
"instance_configuration" : "aws.data.highstorage.d2",
"region" : "ap-southeast-1"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "filter",
"decision" : "NO",
"explanation" : """node does not match index setting [index.routing.allocation.require] filters [data:"hot"]"""
},
{
"decider" : "awareness",
"decision" : "NO",
"explanation" : "there are too many copies of the shard allocated to nodes with attribute [availability_zone], there are [2] total configured shard copies for this shard id and [2] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
}
]
},
{
"node_id" : "",
"node_name" : "",
"transport_address" : "",
"node_attributes" : {
"logical_availability_zone" : "zone-1",
"server_name" : "",
"availability_zone" : "ap-southeast-1b",
"xpack.installed" : "true",
"data" : "hot",
"instance_configuration" : "aws.data.highio.i3",
"region" : "ap-southeast-1"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "disk_threshold",
"decision" : "NO",
"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [9.611063003540039%]"
}
]
},
{
"node_id" : "",
"node_name" : "",
"transport_address" : "",
"node_attributes" : {
"logical_availability_zone" : "zone-0",
"server_name" : "*****",
"availability_zone" : "",
"xpack.installed" : "true",
"data" : "hot",
"instance_configuration" : "aws.data.highio.i3",
"region" : "ap-southeast-1"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.monitoring-alerts-7][0], node[], [P], s[STARTED], a[id=**]]"
},
{
"decider" : "disk_threshold",
"decision" : "NO",
"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [12.751328945159912%]"
},
{
"decider" : "awareness",
"decision" : "NO",
"explanation" : "there are too many copies of the shard allocated to nodes with attribute [availability_zone], there are [2] total configured shard copies for this shard id and [2] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
}
]
}
]
}

8

See here:

The shards can't be allocated (at least not with the current constraints from the allocation attributes because the relevant nodes don't have enough free disk space. You'll have to fix that so recovery of these indices can start.

9

ok let me check with my infra team , will get back to you with results

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.