Logstash redundancy and availability in ELK cluster

Elastic Stack Logstash
1

Hi all,

The question is about Logstash redundancy and availability.

We have an Elasticsearch Cluster running fine with 3 CentOS nodes.

One of this nodes hosts Losgstash. Logstash is set to ingest log files sent by SFTP, at the end, Kibana provide dashboards to show logs info.

I thought, as we have an integrated stack, that an ELK cluster would be able to handle Logstash redundancy, I mean:

  • Logstash configuration replication from one node to others
  • Logstash availability, one node active and other(s) passive, being able to take over in case of first node failure.

The aim is to have only one instance ingesting logs at the same time in order to avoid data duplication. We don't need load-balancing but only redundancy.

Up to now, I’ve not found the way to do it from ELK settings.
Is it possible or no or must I do it with usual HA solutions?

2

Logstash configuration replication from one node to others

This feature is called Centralized Logstash pipeline management, but it is only available if you are using a paid license.

If you are using the free basic license it is not possible, you will need to rely on third-party tools to automate the configuration replication and management.

Logstash availability, one node active and other(s) passive, being able to take over in case of first node failure.

All logstash nodes are independent from each other, there is no native active/passive deployment, you need to user a third-party tool like nginx or HAProxy to do that.

3

Thanks for your answer, I'll handle it with a third-party tool, so.