How to properly connect a standalone Logstash to an Elasticsearch cluster

Elastic Stack Logstash
1

Hello,
I have successfully created an Elasticsearch Cluster with two servers Ubuntu each one running an Elasticsearch node (one ES-Node1 acting as master node and ES-Node2 acting as data node).
I than created a Standalone instance of Logstash running on the same server were ES-Node2 is running and I have specified the following in its output plugin_

output {
    if [tags][0] == "pipeline" or [tags] == "pipeline" {
        elasticsearch {
            index => "zoa-%{sourceType}-%{host}-%{+YYYYMMdd}"
            pipeline => "%{sourceType}"
            **hosts => "xxxx.xxxx.xxxx.xxxx:9200"**
        }
    } else {
        elasticsearch {
            index => "zoa-%{sourceType}-%{host}-%{+YYYYMMdd}"
            **hosts => "xxxx.xxxx.xxxx.xxxxx:9200"**
            }
    }
file {
      path => "/tmp/test_output_logstash.log"
      }
}

where xxxx.xxxx.xxxx designates the ip address of the host where ES-Node2.
I than run three different tests by sending some data from Logstash to Elasticsearch cluster
Test1
Both ES-Node1 and ES-Node2 are up and running
ES nodes receive and index the data being sent (I have used command curl http://localhost:9200/_cat/indices to verify that)
Test2
ES-Node1 is up and running while ES-Node2 has been brought down
No data are received and I get a connection problem with Logstash with ES (that can be expected because of one of the two node is inactive). I would expect that as this is a Cluster having one node active is enough to get the data...
Test3
ES-Node2 is up and running while ES-Node1 has been brought down
No data are received and I get a connection problem with Logstash with ES (that can be expected because of one of the two node is inactive). I would expect that as this is a Cluster having one node active is enough to get the data...

So I a m fine with Test1 results but not with Test2 and Test3 as I expect that having at least one node of the cluster active is enough to get the data.....
Is my Logstash configuration setting properly specified?
Thanks and bye

2

I don't do elasticsearch much but my understanding is that the cluster needs both master and data nodes. If either is down then the cluster cannot operate. If you want to be able to run with one of the nodes down then do not specify node.roles and let each be general purpose.

3

No, this is not how elasticsearch works, for the cluster to work it needs a healthy and up master node.

To have any kind of resilience you need at least a 3 nodes cluster where both nodes are master and data nodes, in this scenario your cluster will still works if you lose one node, if you lose two nodes, your cluster won't work until the nodes are brought back.

With a 2 node cluster only one can be a master eligible node, if the master eligible node is also a data node, your cluster will still runs if you lose the data only node, but if you lose the master node it will not work.

From what you described you have Es-Node1 acting as a master only and Es-Node2 acting as a data node only, your cluster will only work when both nodes are up because you need a up and running master and a up and running data node.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.