One of our logstash servers suddenly essentially sropped processing our logs and started displaying the log entries mentioned in the title.
The index in question appears to be ok according to elasticsearch.
Both are version 5.4.
A relevant bit from the log:
[2018-05-08T15:43:47,776][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 503 ({"type"=>"unavailable_shards_exception", "reason"=>"[logstash-2018.05.06][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[logstash-2018.05.06][0]] containing [6] requests]"})
[2018-05-08T15:43:47,776][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>6}
The health of the index in question:
logstash-2018.05.06 2 p STARTED 27 89kb ip.ip.ip.ip data-node-1
logstash-2018.05.06 1 p STARTED 30 93.4kb ip.ip.ip.ip data-node-2
logstash-2018.05.06 0 p STARTED 24 101.3kb ip.ip.ip.ip data-node-3
Any guidance would be appreciated.