Cluster Red - unallocated shards in a index

Elastic Stack Elasticsearch
1

My cluster suddenly went to red. Because of an index shard allocation fail. when i run

GET /_cluster/allocation/explain
{
  "index": "my_index",
  "shard": 0,
  "primary": true
}

output:

{
  "shard": {
    "index": "twitter_tracker",
    "index_uuid": "mfXc8oplQpq2lWGjC1TxbA",
    "id": 0,
    "primary": true
  },
  "assigned": false,
  "shard_state_fetch_pending": false,
  "unassigned_info": {
    "reason": "ALLOCATION_FAILED",
    "at": "2018-01-02T08:13:44.513Z",
    "failed_attempts": 1,
    "delayed": false,
    "details": "failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: NotSerializableExceptionWrapper[shard_lock_obtain_failed_exception: [twitter_tracker][0]: obtaining shard lock timed out after 5000ms]; ",
    "allocation_status": "no_valid_shard_copy"
  },
  "allocation_delay_in_millis": 60000,
  "remaining_delay_in_millis": 0,
  "nodes": {
    "n91cV7ocTh-Zp58dFr5rug": {
      "node_name": "elasticsearch-24-384-node-1",
      "node_attributes": {},
      "store": {
        "shard_copy": "AVAILABLE"
      },
      "final_decision": "YES",
      "final_explanation": "the shard can be assigned and the node contains a valid copy of the shard data",
      "weight": 0.45,
      "decisions": []
    },
    "_b-wXdjGRdGLEtvY76PDSA": {
      "node_name": "elasticsearch-24-384-node-2",
      "node_attributes": {},
      "store": {
        "shard_copy": "NONE"
      },
      "final_decision": "NO",
      "final_explanation": "there is no copy of the shard available",
      "weight": 0,
      "decisions": []
    }
  }
}

What will be the solution? This is happened in my production node.

2

In the final decision, it shows that the data already available. So how can I get back my shard (reallocation)

3

I solved my issue with the following command.

curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -d '{
    "commands" : [ {
        "allocate_stale_primary" :
            {
              "index" : "da-prod8-other", "shard" : 3,
              "node" : "node-2-data-pod",
              "accept_data_loss" : true
            }
        }
    ]
}'

So here u might lose the data. For me it works well. It was thrilling while running the command. luckily it worked well.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.