# How to achieve Clusters in Logstash?

**URL:** <https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387>\
**Category:** Logstash\
**Created:** [November 20, 2017, 1:16pm UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387 "2017-11-20T13:16:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshay](https://avatars.discourse-cdn.com/v4/letter/a/3ab097/32.png) [@akshay](https://discuss.elastic.co/u/akshay)\
**Post date:** [November 20, 2017, 1:16pm UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/1 "2017-11-20T13:16:36Z")

</div>

Hello,  
Following is the deployment scenario.  
We are using ELK stack (ElasticSearch,Logstash and Kibana for UI)  
Elastic Search will be deployed on 3 nodes which form one Cluster.  
We also want to Cluster Logstash which provides Input for ES.  
Presently we are thinking of deploying 2 Logstash Instances in Cluster.  
Can you suggest how this can be achieved?  
There are 4 FileBeat instances per Logstash.  
Pl. let me know.  
Thanks in adavnce....

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 20, 2017, 2:38pm UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/2 "2017-11-20T14:38:41Z")

</div>

Logstash doesn't have any native clustering functionality. What are you trying to achieve?

---

<div class="post-metadata">

**Author:** ![akshay](https://avatars.discourse-cdn.com/v4/letter/a/3ab097/32.png) [@akshay](https://discuss.elastic.co/u/akshay)\
**Post date:** [November 21, 2017, 4:51am UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/3 "2017-11-21T04:51:08Z")

</div>

Following is the deployment scenario.  
We are using ELK stack (ElasticSearch,Logstash and Kibana for UI)  
Elastic Search will be deployed on 3 nodes which form one Cluster.  
We also want to Cluster Logstash which provides Input for ES.  
Presently we are thinking of deploying 2 Logstash Instances in Cluster.  
Can you suggest how this can be achieved?  
There are 4 FileBeat instances per Logstash.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 21, 2017, 6:18am UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/4 "2017-11-21T06:18:28Z")

</div>

**Why** do you want to "cluster" Logstash? What's the end goal? Better fault tolerance? Increased performance? Both? What kind of inputs do you have? How many events per second do you expect to process?

---

<div class="post-metadata">

**Author:** ![akshay](https://avatars.discourse-cdn.com/v4/letter/a/3ab097/32.png) [@akshay](https://discuss.elastic.co/u/akshay)\
**Post date:** [November 21, 2017, 8:05am UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/5 "2017-11-21T08:05:59Z")

</div>

Tell me one thing if I have 8 file beat instance with me & my logstash input configuration is like,  
input {  
beats {  
host =\> host1  
port =\> port1  
}  
beats {  
host =\> host2  
port =\> port2   
}  
beats {  
host =\> host3  
port =\> port3  
}  
.......

beats {  
host =\> hostn  
port =\> portn  
}  
}

& if my logstash instance get crashed or in ideal mode then how do I recognize it? & In this process if some data lost happend then how do I track all those data?  
Is there any recovery mechanism logstash have using we identify above scenarios?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 21, 2017, 8:21am UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/6 "2017-11-21T08:21:23Z")

</div>

You've probably misunderstood what the `host` option does. I don't think you need more than one beats input.

> if my logstash instance get crashed or in ideal mode then how do I recognize it?

You could e.g. connect Logstash to Lovebeat or a similar tool for heartbeat monitoring.

> In this process if some data lost happend then how do I track all those data?

With Logstash's persistent queue functionality and Filebeat's way of backing off when Logstash is unable to accept events I don't see why you'd lose data (for reasonably short outages at least).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2017, 8:21am UTC](https://discuss.elastic.co/t/how-to-achieve-clusters-in-logstash/108387/7 "2017-12-19T08:21:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.