We are deliberating the feasibility of adding Confluent Kafka as a message broker to our Elastic Search instance.
Now that logstash comes with its own persistence and dead letter queues, I have some questions in this regard.
(1) Having a message queue layer like Kafka decouples the beat and logstash services which allows for handling work load bursts. How does the built-in queues of logstash handle this ?
(2) Is there any negative impact on Elastic Search performance with adding a message queue layer ?
Senthil Nathan M