Cluster ( or High Avalibility) at Logstash level

[Yashwant_Shettigar]

How to achieve Clusters in Logstash?

Hello, Following is the deployment scenario. We are using ELK stack (ElasticSearch,Logstash and Kibana for UI) Elastic Search will be deployed on 3 nodes which form one Cluster. We also want to Cluster Logstash which provides Input for ES. Presently we are thinking of deploying 2 Logstash Instances in Cluster. Can you suggest how this can be achieved? There are 4 FileBeat instances per Logstash. Pl. let me know. Thanks in adavnce....

Hi Team,

I need to know how can I achieve HA(High Availability) at logstash level (2 logstash servers)

Reason: When we route logs from Firewall devices, Storage devices, etc we can define only IP address for log management server. It is not possible to add more than one IP address.

At beat level there is no issue.

Thank You

[elasticforme (Sachin)]

logstash itself is not HA. if you really want HA setup you have to introduce kafka or some kind of other layer in middle.

[Yashwant_Shettigar]

Thanks Sachin.

Can you help me with some guide which will help me in setting this up ? Because most of the guides are showing combination of beats and kafka. I want something which is independent, which can be used only for storage and firewall devices.

Also, apart from kafka, is there anything else that can help ?

Thank You

[elasticforme (Sachin)]

Yashwant
sorry I have no experience in kafka setup.

I only work with logstash for now.