I have a bunch of hosts that im monitoring with logstash and they all send data to my elastic cluster.
When the clusters health turn red for reasons, some of the logstash instances seem to die and stop sending
data back to the cluster, even after its turned green (from red).
Here is a pastebin of the log output.
The output is restored if i
killall -9 -u logstash and
service logstash start.
Here is a graph that shows the problem.
Is there a setting i can set to stop re-trying stuff for ever as its mentioned here ?
Or is there anything else i can do for the failing servers to be able to self-recover ?
Im running all the latest and greatest ELK stuff.