Elasticsearch Integration with Kafka, Zookeeper using filebeat

[gauravbhutani30 (Gaurav Bhutani)]

Hi Everyone,

I am have a Elastic cluster running with nodes spread across multiple datacenters.
Now, I have to ensure the data synchronization across the nodes. I came across Kafka which is used to process large data and it uses Zookeeper to sync the data between the nodes.

Just Enough Kafka for the Elastic Stack, Part 1

Can you please help me with the step by step guide on how to install, configure and run kafka with Elastisearch using filebeat as suggested in the above link.

I found the below link but did not get any practical example to configure it:
https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html

Please help and suggest if there is any other solution!!

[steffens (Steffen Siering)]

The article wants you to send filebeat->logstash->kafka. In case you don't want any other inputs for logstash, you can have filebeat directly send to kafka.

Here's a sample how to have filebeat send data to logstash.

check out filebeat.full.yml or the filebeat docs for all output settings.