Sending Data to DataNodes / ClientNodes

Elastic Stack Elasticsearch
1

I have 4 nodes with 3 master and 3 data nodes with default shard/replica setting as below ,

  1. Node1 - master =true,data=false
  2. Node1 - master =true,data=true
  3. Node1 - master =true,data=true
  4. Node1 - master =false,data=true

I am able to send data through Logstash to one of the data node , I have couple of questions.

  1. Instead of passing the data directly to datanode can i pass it to somekind of cluster aware client node that will just load balance and send it to a datanode.

  2. If the above cannot be done then should i have multiple outputs in my logstash conf file for specific log type to be sent to specific datanode.

Thanks

2

You can use a client node for indexing data.
Just set master=false and data=false.

This node is aware of the cluster and will do all the loadbalancing for you! :slight_smile:

Logstash v2.x can be configured to load balance on multiple client nodes