How to solve unavailable_shards_exception?

Elastic Stack Elasticsearch
1

I have encountered the following exception

[2017-07-13T16:51:02,186][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-im_message-2017.07.13][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-im_message-2017.07.13][2]] containing [4] requests]"})
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Retrying individual actions
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action

There is an unassigned shard, I have no idea if this is the cause,

{
  "index": "logstash-im_message-2017.07.13",
  "shard": 2,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "INDEX_CREATED",
    "at": "2017-07-13T00:05:03.861Z",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "bts3Wo5eQ62lmhBLUH022g",
      "node_name": "spiderman",
      "transport_address": "192.168.0.213:9300",
      "node_decision": "no",
      "weight_ranking": 1,
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id": "75Uk20LHSSC4aV6lD0nlBQ",
      "node_name": "thor",
      "transport_address": "192.168.0.214:9300",
      "node_decision": "no",
      "weight_ranking": 2,
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id": "bE78po56TuatkgI-Ijq29w",
      "node_name": "hulk",
      "transport_address": "192.168.0.215:9300",
      "node_decision": "no",
      "weight_ranking": 3,
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    }
  ]
}
2

Having same problem here (Version 5.1.1 for Logstash and Elasticsearch). The services are up with same error in Logstash log:
[logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[filebeat-2017.07.14][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [filebeat-2017.07.14] containing [44] requests]"})
And everything looks fine in Elasticsearch log though.
Looking forward to solution.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.