Should Master nodes only receive the traffic in an ES cluster?

Elastic Stack Elasticsearch
1

Currently, I have 16 nodes (5 master + 11 data nodes) in my cluster and via Haproxy I am sending the requests to only the master nodes. Recently, due to some issue one of the master node was flapping and latency spiked for the whole cluster.
Questions:

  1. Should the request be routed only to the master nodes?
  2. what is the adviced number of master nodes for a production cluster?
  3. Should I have dedicated coordinating nodes for this task?
2

Hello and welcome,

No, if you have dedicated master nodes, you should avoid sending requests to them, use the data nodes of your cluster, if you have data tiering in your cluster, like some hot data nodes and some warm or cold data nodes, use the hot data nodes.

It is a best practice to limit the number of master-eligible nodes in the cluster to 3.

Also no, coordinating nodes are rarely used/recommended these days, specially for such small cluster.

Just use the data nodes for your indexing and query requests.

3

@leandrojmp
Wouldn't directing traffic directly to the data nodes increase the chances of OOM (out of memory) issues on the node?
We do perform scroll, bulk edit and bulk fetch operations on the cluster

4

I don't think so, this is the recommended approach, send your indexing and query request to your data nodes, when using data tiering, send them to the hot nodes.