Unavailable_shards_exception primary shard is not active

Elastic Stack Logstash
1

One of our logstash servers suddenly essentially sropped processing our logs and started displaying the log entries mentioned in the title.

The index in question appears to be ok according to elasticsearch.
Both are version 5.4.

A relevant bit from the log:
[2018-05-08T15:43:47,776][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-2018.05.06][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-2018.05.06][0]] containing [6] requests]"})
[2018-05-08T15:43:47,776][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>6}

The health of the index in question:
logstash-2018.05.06 2 p STARTED 27 89kb ip.ip.ip.ip data-node-1
logstash-2018.05.06 1 p STARTED 30 93.4kb ip.ip.ip.ip data-node-2
logstash-2018.05.06 0 p STARTED 24 101.3kb ip.ip.ip.ip data-node-3

Any guidance would be appreciated.

2

Ignore this, was looking in the wrong place.
It was outputting to two elasticsearch clusters and one of them had the broken index.
All is well now.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.