Logstash HA

Elastic Stack Logstash
1

Hi Team,

I am using logstash component for collecting/transforming data and then send to pubsubLite. Now there are cases to collect data using push mechanism (other sources are pushing data to logstash component) as well as pull mechanism (logstash is calling data or event through API/ python script).
Can you please guide/suggest way to achieve HA in logstash while pulling the data.

2

I still looking for good HA solution, pls update

3

There's historic threads on this that still apply - there's nothing native and you will need a load balancer.

1 Like
4

If you are using multiple logstash instances to pull the same data from an API and want to avoid duplicates in Pub/Sub Lite that is really not a logstash question. If the output were elasticsearch then you could set the document_id to a hash of the message fields using a fingerprint filter, which would cause duplicates to overwrite oneanother, but I do not know if Pub/Sub provides a way to de-duplicate data by using a unique message id or some other method.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.