Hey guys
I need some advice on an ELK setup we have at work. We have 3 ElasticSearch (6.x) nodes (all mdi), and 3 Kibana (6.x) instances on the same boxes as ES. Then we have 2 Logstash instances (each on it's own node, not on the ES nodes) configured with RabbitMQ inputs, to consume from the same rabbit queue and publish to ElasticSearch. While this works most of the time, it seems that every now and then the logstash rabbit queue (prefetch 1000) backs up, for a few minutes, sometimes it seems like it doesn't recover, and then I bounce each logstash instance to fix the issue.
Can anyone point me in the right direction? I've noticed that if I run a Kibana query (e.g get all data for the last 24 hours), then I can sometimes reproduce the issue. But it's not always the case.
Thanks
Renault