Logstash and redis High Availability

Elastic Stack Logstash
1

I have currently 2 Redis instances in active active modes using Redhat cluster and i'm feeding them with log shippers,
We have 2 other servers with Logstash installed by which events would be popped up from both redis servers and stored into ElasticSearch.

Is it possible to to tell Logstash1 to pop up events from Redis1 and if Redis1 isn't available pop up events from Redis2 and vice versa????

Thanks

2

hello

3

I don't believe you can do this in Logstash itself. You would need to setup something like HAProxy that can check for up/down states and send the request somewhere based on hostname. I think that should get you where you want to go.

4

Thanks, instead of HAproxy, i'm using pacemaker (redhat cluster), both redis are in active active and being monitored by pacemaker itself, however, the issue is, logstash1 is popping event from redis1 and logstash2 is popping event from redis2. i needed a directive by which i can tell logstash pop up event from redis1 and if not available pop up event from redis2.
thanks man, much appreciated

5

Kafka is very well suited for a distributed highly available queue. Check
it out if you haven't.

1 Like