IP based failover in Logstash

[Debarati_Goswami]

Hi All ,

I have a requirement to collect logs from various sources into Logstash cluster (of 6 nodes) of which 4 nodes are for 4 different types of logs (rsyslog , IPFIX , SFTP and FluentD) and the last two nodes would work as dedicated backup nodes for Rsyslog and IPFIX in case of original node failure . My main problem here is that we are not provisioned with LoadBalancer , hence I am trying to find ways for IP based failover in Logstash to facilitate load movement from one node to another in case of Node failure.

I am absolutely new to ELK and any help would be extremely helpful for me.

[Badger]

You can have two logstash instance listen on different IPs. How you would tell whatever is sending data to them to fail over from one to the other is a question about that software, not logstash.

[leandrojmp (Leandro Pereira)]

This is related to your infrastructure, logstash instances are independent from each other, so to implement a load balancing or failover mechanism you will need other tools.

In your case I recommend that you look at keepalived, you could create a VIP IP that would float between the two nodes according to some checks that you will need to implement, for example if the logstash service stops, keepalived would change the VIP IP to the other node.

But how to configure it is out of the scope of this forum.

[elasticforme (Sachin)]

We have tested HA Proxy. basically your client will send data to x.x.x.x:port and that will be proxy ip:port

from there Proxy will send data to either node1:port and node2:port

if any of the node goes down all the data goes to one node and you have fail over.
and when both logstash nodes are up they are load balanced.

on both logstash node you run same pipeline same config.