To which elasticsearch node should logstash send to

Elastic Stack Elasticsearch
1

Hi all,

A little background about my production test-bed before my question:

Elasticsearch 6.0:

  • 3 nodes, both data and master-eligible nodes. Each with 16G of RAM, with JVM heap set to 8G.
  • 1 node, coordinating-only node with 4G . JVM heap set to 2G. Kibana runs on this node.

Logstash 6.0:

  • 2 nodes, running only Logstash, each 4G of RAM.

Questions:

  1. To which ES node(s) should each logstash have the output set to?
  2. Is the coordinating-only node JVM heap fairly ok?
2

As per your configuration, if you have coordinating node then it would be better to send the logs to this node.
Coordinating node is used for load balancing and handling request . whenever any request comes it redirect to appropriate data node and aggregate the response from nodes and send back to the application.
As all nodes are in a cluster , data will be balanced all nodes. For more documentation refer this: Node | Elasticsearch Guide [8.11] | Elastic

To which ES node(s) should each logstash have the output set to?
logstash's output should set to Coordinating node in ES cluster.

Is the coordinating-only node JVM heap fairly ok?
As coordinating node only handle the request not ingest the data and process it so it would be fine.

3

Thanks, Pooja.

I had already read gone through the link that you referenced. That's why I had Kibana installed on the coordinating-only node. But, it wasn't clear to me whether the same node could be used for receiving data from logstash.

Question:

In this current set-up of mine, having only 1 coordinating-only node to receive from Logstash, makes the setup have a single point of failure. Hence, would it be taxing or helping the cluster if I add another coordinating-only node?

4

But, it wasn't clear to me whether the same node could be used for receiving data from logstash.
yes same node can be used as output for logstash.

In this current set-up of mine, having only 1 coordinating-only node to receive from Logstash, makes the setup have a single point of failure. Hence, would it be taxing or helping the cluster if I add another coordinating-only node?
As now you are aware of working of coordinating node . i.e. it is used for load -balancing and used for handling the search request across the cluster. in case , this node is down then we can recover via using master eligible node .
While master nodes can also behave as coordinating nodes and route search and indexing requests from clients to data nodes, it is better not to use dedicated master nodes for this purpose. It is important for the stability of the cluster that master-eligible nodes do as little work as possible.

To create a standalone master-eligible node, set:

node.master: true
node.data: false
node.ingest: false

doing this we can not face any data loss problem and also single point of failure.

5

You have a good set-up with 3 master eligible nodes. Heap sizes also seem reasonable, but this will depend on your load and how much data and shards you aim to hold in the cluster.

For this kind of setup I would recommend having Logstash send requests to the master/data nodes directly. This will avoid overloading the coordinating only node.

1 Like
6

Sorry, your reply is not coherent to my question.

8

hi Christian,

Yes, I do understand that it all depends upon the load. I just wanted to ascertain that this was practically and theoretically a good enough set-up to begin testing.

Thanks.

9

As you asked about adding new coordinating node in ES cluster in that case i would recommend you set a node as master eligible node.
As i told earlier master nodes can also behave as coordinating nodes and route search and indexing requests from clients to data nodes, it is better not to use dedicated master nodes for this purpose. It is important for the stability of the cluster that master-eligible nodes do as little work as possible.

if this node getting failure then other master node will handle the request which is also a part of your cluster.

for adding a new coordinating node you can do it . make sure it is also a part of your cluster and logstash send the request to both nodes and if one gets fail then request is handled by another node.

If your query still not answered request you to please acknowledge .

10

Sorry, I've some additional questions come up during the process:

  • I need to enable x-pack for the cluster. Should I install it on all the 6 nodes of the cluster?
11

Yes, X-Pack need to be installed on all nodes in the cluster.

1 Like
12

Thanks @Christian_Dahlqvist

13

Thanks for the help! :slight_smile: My query was answered in one of the comments; I've marked it as the solution.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.