I have encountered the following exception
[2017-07-13T16:51:02,186][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-im_message-2017.07.13][2] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-im_message-2017.07.13][2]] containing [4] requests]"})
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Retrying individual actions
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
[2017-07-13T16:51:02,186][ERROR][logstash.outputs.elasticsearch] Action
There is an unassigned shard, I have no idea if this is the cause,
{
"index": "logstash-im_message-2017.07.13",
"shard": 2,
"primary": false,
"current_state": "unassigned",
"unassigned_info": {
"reason": "INDEX_CREATED",
"at": "2017-07-13T00:05:03.861Z",
"last_allocation_status": "no_attempt"
},
"can_allocate": "no",
"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions": [
{
"node_id": "bts3Wo5eQ62lmhBLUH022g",
"node_name": "spiderman",
"transport_address": "192.168.0.213:9300",
"node_decision": "no",
"weight_ranking": 1,
"deciders": [
{
"decider": "replica_after_primary_active",
"decision": "NO",
"explanation": "primary shard for this replica is not yet active"
},
{
"decider": "throttling",
"decision": "NO",
"explanation": "primary shard for this replica is not yet active"
}
]
},
{
"node_id": "75Uk20LHSSC4aV6lD0nlBQ",
"node_name": "thor",
"transport_address": "192.168.0.214:9300",
"node_decision": "no",
"weight_ranking": 2,
"deciders": [
{
"decider": "replica_after_primary_active",
"decision": "NO",
"explanation": "primary shard for this replica is not yet active"
},
{
"decider": "throttling",
"decision": "NO",
"explanation": "primary shard for this replica is not yet active"
}
]
},
{
"node_id": "bE78po56TuatkgI-Ijq29w",
"node_name": "hulk",
"transport_address": "192.168.0.215:9300",
"node_decision": "no",
"weight_ranking": 3,
"deciders": [
{
"decider": "replica_after_primary_active",
"decision": "NO",
"explanation": "primary shard for this replica is not yet active"
},
{
"decider": "throttling",
"decision": "NO",
"explanation": "primary shard for this replica is not yet active"
}
]
}
]
}