Need help with Kafka and Logstash with elastic Search cluster

[Blason (R)]

HI Guys,

I need a help on my setup mentioned below. Can someone please share some insight [ I tried searching a lot through previous discussions but unable to get the exact details hence posting it here]

I have around 5-6k Nodes/devices/servers - I am planning to use

1 Logstash instance
3 ES instances [ one dedicated master node, other data as well as master]
1 Kibana

My main concerns as as below

1. Where do I configure kafka broker? On logstash?
2. Is there any good information available or How to? for integrating logstash with Kafka
3. How does my single instance of logstash connects to ES cluster?
4. Are there any other steps that I need to perform for data resiliency?

[yaauie (Ry Biesemeyer)]

Logstash is a data pipeline engine that allows you to configure one or more data pipelines, each of which has at least one input, zero or more filters, and at least one output.

If you have a kafka topic that you wish to subscribe to, I suspect you would create a Logstash pipeline that looks something like this:

input {
  kafka {
    # kafka connection config
    # see: https://www.elastic.co/guide/en/logstash/6.7/plugins-inputs-kafka.html
  }
}
filter {
  # one or more filters to mutate and/or enrich the events
  # see: https://www.elastic.co/guide/en/logstash/6.7/filter-plugins.html
}
output {
  elasticsearch {
    # elasticsearch connection config
    # see: https://www.elastic.co/guide/en/logstash/6.7/plugins-outputs-elasticsearch.html
  }
}

[Blason (R)]

I see and what does my output should look like if I have cluster? It should send logs to dedicated master?

[yaauie (Ry Biesemeyer)]

Typically you would list several nodes in a cluster, which may or may not be masters. The output will balance the requested between them, and each Elasticsearch node will route the requests to the best node to handle the indexing.