# Need help with Kafka and Logstash with elastic Search cluster **URL:** \ **Category:** Logstash\ **Created:** [May 9, 2019, 5:36am UTC](https://discuss.elastic.co/t/need-help-with-kafka-and-logstash-with-elastic-search-cluster/180280 "2019-05-09T05:36:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [May 9, 2019, 5:36am UTC](https://discuss.elastic.co/t/need-help-with-kafka-and-logstash-with-elastic-search-cluster/180280/1 "2019-05-09T05:36:28Z")
HI Guys, I need a help on my setup mentioned below. Can someone please share some insight [I tried searching a lot through previous discussions but unable to get the exact details hence posting it here] I have around 5-6k Nodes/devices/servers - I am planning to use 1 Logstash instance 3 ES instances [one dedicated master node, other data as well as master] 1 Kibana My main concerns as as below 1. Where do I configure kafka broker? On logstash? 2. Is there any good information available or How to? for integrating logstash with Kafka 3. How does my single instance of logstash connects to ES cluster? 4. Are there any other steps that I need to perform for data resiliency? ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [May 9, 2019, 5:40pm UTC](https://discuss.elastic.co/t/need-help-with-kafka-and-logstash-with-elastic-search-cluster/180280/2 "2019-05-09T17:40:50Z")
Logstash is a data pipeline engine that allows you to configure one or more data pipelines, each of which has at least one input, zero or more filters, and at least one output. If you have a kafka topic that you wish to subscribe to, I suspect you would create a Logstash pipeline that looks something like this: ```auto input { kafka { # kafka connection config # see: https://www.elastic.co/guide/en/logstash/6.7/plugins-inputs-kafka.html } } filter { # one or more filters to mutate and/or enrich the events # see: https://www.elastic.co/guide/en/logstash/6.7/filter-plugins.html } output { elasticsearch { # elasticsearch connection config # see: https://www.elastic.co/guide/en/logstash/6.7/plugins-outputs-elasticsearch.html } } ``` ---
**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\ **Post date:** [May 10, 2019, 3:32am UTC](https://discuss.elastic.co/t/need-help-with-kafka-and-logstash-with-elastic-search-cluster/180280/3 "2019-05-10T03:32:09Z")
I see and what does my output should look like if I have cluster? It should send logs to dedicated master? ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [May 10, 2019, 5:01am UTC](https://discuss.elastic.co/t/need-help-with-kafka-and-logstash-with-elastic-search-cluster/180280/4 "2019-05-10T05:01:30Z")
Typically you would list several nodes in a cluster, which may or may not be masters. The output will balance the requested between them, and each Elasticsearch node will route the requests to the best node to handle the indexing. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 7, 2019, 5:12am UTC](https://discuss.elastic.co/t/need-help-with-kafka-and-logstash-with-elastic-search-cluster/180280/5 "2019-06-07T05:12:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.