Does someone knows how to scale Logstash? What do I mean by scaling? Let's say I have one instance of Logstash producing messages from jdbc to Kafka and lets say that node where Logstash service is installed and instance of him is running fall down suddenly.
How to continue with the same process, something like master-slave config? Is there any way to acomplish that or I need to set some work around by myself?
The jdbc input doesn't support sharing its state file with another instance, so there's no official support for having two instances running at the same time if they're using a jdbc input.
However, if you can verify that the instance is down, you can point another Logstash instance to the same state file and have it continue roughly at the same place where the first instance stopped.
I wouldn't call this "scaling", by the way.
Is there anything implemented to "heartbeat" the logstash instances or I need to implemented by myself? Any suggestions? Maybe firebeat?
And thank you for the response 
What kind of heartbeat do you need? Recent versions of Logstash has a monitoring API, or you could use an output plugin of your choice to ping something for each processed event. Lovebeat can help you figure out when heartbeats have stopped arriving.
See also https://www.elastic.co/blog/how-to-check-logstashs-pulse.
Something like kafka group managment. If I quote you "if you can verify that the instance is down, you can point another Logstash instance to the same state file and have it continue roughly at the same place where the first instance stopped.". I need exactly this
And there's no built-in support for that.
You can suggest me something?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.