How to achieve Clusters in Logstash?

Elastic Stack Logstash
1

Hello,
Following is the deployment scenario.
We are using ELK stack (ElasticSearch,Logstash and Kibana for UI)
Elastic Search will be deployed on 3 nodes which form one Cluster.
We also want to Cluster Logstash which provides Input for ES.
Presently we are thinking of deploying 2 Logstash Instances in Cluster.
Can you suggest how this can be achieved?
There are 4 FileBeat instances per Logstash.
Pl. let me know.
Thanks in adavnce....

2

Logstash doesn't have any native clustering functionality. What are you trying to achieve?

3

Following is the deployment scenario.
We are using ELK stack (ElasticSearch,Logstash and Kibana for UI)
Elastic Search will be deployed on 3 nodes which form one Cluster.
We also want to Cluster Logstash which provides Input for ES.
Presently we are thinking of deploying 2 Logstash Instances in Cluster.
Can you suggest how this can be achieved?
There are 4 FileBeat instances per Logstash.

4

Why do you want to "cluster" Logstash? What's the end goal? Better fault tolerance? Increased performance? Both? What kind of inputs do you have? How many events per second do you expect to process?

5

Tell me one thing if I have 8 file beat instance with me & my logstash input configuration is like,
input {
beats {
host => host1
port => port1
}
beats {
host => host2
port => port2
}
beats {
host => host3
port => port3
}
.......

beats {
host => hostn
port => portn
}
}

& if my logstash instance get crashed or in ideal mode then how do I recognize it? & In this process if some data lost happend then how do I track all those data?
Is there any recovery mechanism logstash have using we identify above scenarios?

6

You've probably misunderstood what the host option does. I don't think you need more than one beats input.

if my logstash instance get crashed or in ideal mode then how do I recognize it?

You could e.g. connect Logstash to Lovebeat or a similar tool for heartbeat monitoring.

In this process if some data lost happend then how do I track all those data?

With Logstash's persistent queue functionality and Filebeat's way of backing off when Logstash is unable to accept events I don't see why you'd lose data (for reasonably short outages at least).

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.