Logstash HA

Hi Team,

I am using logstash component for collecting/transforming data and then send to pubsubLite. Now there are cases to collect data using push mechanism (other sources are pushing data to logstash component) as well as pull mechanism (logstash is calling data or event through API/ python script).
Can you please guide/suggest way to achieve HA in logstash while pulling the data.

I still looking for good HA solution, pls update

There's historic threads on this that still apply - there's nothing native and you will need a load balancer.

1 Like

If you are using multiple logstash instances to pull the same data from an API and want to avoid duplicates in Pub/Sub Lite that is really not a logstash question. If the output were elasticsearch then you could set the document_id to a hash of the message fields using a fingerprint filter, which would cause duplicates to overwrite oneanother, but I do not know if Pub/Sub provides a way to de-duplicate data by using a unique message id or some other method.