Kafka with elk stack


(Jay Dabhi) #1

Hi,

When should we use kafka in ELK stack?
Where should we put kafka module in ELK stack provided filebeat is being used for log shipping?


(Mark Walkom) #2

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out :wink:

Usually Kafka is deployed as a broker layer, see https://www.elastic.co/guide/en/logstash/5.6/deploying-and-scaling.html#integrating-with-messaging-queues for some info.


(Jonar B) #3

are there some guides on how to deploy and configure with this architecture?
I already tried to use kafka but on my producer side, it does not send data to elasticsearch.


(Mark Walkom) #4

Kafka does not send to Elasticsearch anyway, you still need Logstash there to extract from it and then do the processing and sending.


(Jonar B) #5

Yes @warkolm, that's also what i did.
ES -> logstash -> kafka -> logstash -> es


(Mark Walkom) #6

Might b be easier to start another thread with more, specific details :slight_smile:


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.