Can we form a cluster for logstash?

[yash_mangla (Yash M.)]

Hello folks,

Is it possible to make a logstash cluster to increase it's capacity in terms of logs processing, transformation and forwarding.

i think we need this kind of setup for larger scale because when we correlating between multiple index of elasticsearch in realtime we need strong processing system to compare each record with millions of available document at same place.

I required your input and feedback as well in this topic. please help me in this context to make better choice for enterprise workflow.

Regards,
Yash M.

[magnusbaeck (Magnus Bäck)]

Logstash instances can't cluster in the same sense Elasticsearch can. Depending on your exact requirements and goals this may or may not be a limitation. It's not clear what kind of correlation you want to perform and how that should be implemented efficiently.

[yash_mangla (Yash M.)]

Hi @magnusbaeck,

We aware about we can make cluster into elasticsearch. but we have required cluster at logstash end the reason is all the processing we are doing with logstash only if it goes down then it's single point of failure for us even we have the elasticsearch cluster.

[magnusbaeck (Magnus Bäck)]

Okay, so that actual requirement is "no single point of failure for Logstash"? Or are there other requirements? You talked about correlations earlier.

[yash_mangla (Yash M.)]

yeah, what if we are performing lookup query for each records with millions of documents in that case i need strong processing speed. so i just looking a solution for it..

[magnusbaeck (Magnus Bäck)]

If you send all inbound events to a message broker you can have multiple Logstash instances pulling from the queue set up in the broker. Then you get both horizontal scalability as well as fault-tolerance.