Kafka with elk stack

(Jay Dabhi) #1


When should we use kafka in ELK stack?
Where should we put kafka module in ELK stack provided filebeat is being used for log shipping?

(Mark Walkom) #2

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out :wink:

Usually Kafka is deployed as a broker layer, see https://www.elastic.co/guide/en/logstash/5.6/deploying-and-scaling.html#integrating-with-messaging-queues for some info.

(Jonar B) #3

are there some guides on how to deploy and configure with this architecture?
I already tried to use kafka but on my producer side, it does not send data to elasticsearch.

(Mark Walkom) #4

Kafka does not send to Elasticsearch anyway, you still need Logstash there to extract from it and then do the processing and sending.

(Jonar B) #5

Yes @warkolm, that's also what i did.
ES -> logstash -> kafka -> logstash -> es

(Mark Walkom) #6

Might b be easier to start another thread with more, specific details :slight_smile:

(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.