Recommendation for number of master nodes in a large cluster having 250 data nodes

[Praveen_Atluri]

Hi All, Is there any recommendation for how many master nodes must be there for a large cluster containing 250 data nodes?
When we monitor the thread pool of each data node in the cluster, I see some tasks that are rejected like below

 "write" : {
          "threads" : 8,
          "queue" : 13,
          "active" : 8,
          "rejected" : 19958,
          "largest" : 8,
          "completed" : 11249210
        }

Does this indicate a problem with ingestion? how do we rectify this issue? I see this in some of the data nodes.

[DavidTurner]

The reference manual has the following recommendation:

However, it is good practice to limit the number of master-eligible nodes in the cluster to three. Master nodes do not scale like other node types since the cluster always elects just one of them as the master of the cluster. If there are too many master-eligible nodes then master elections may take a longer time to complete. In larger clusters, we recommend you configure some of your nodes as dedicated master-eligible nodes and avoid sending any client requests to these dedicated nodes. Your cluster may become unstable if the master-eligible nodes are overwhelmed with unnecessary extra work that could be handled by one of the other nodes.

The master nodes usually have nothing to do with ingestion, they only come into play when creating indices and updating mappings and so on.

[Praveen_Atluri]

Thank you!

When I checked the node stats for some of my nodes, I see below for write thread pool.

 "write" : {
          "threads" : 8,
          "queue" : 13,
          "active" : 8,
          "rejected" : 19958,
          "largest" : 8,
          "completed" : 11249210
        }

Does this indicate that elasticsearch is unable to cope with the rate of ingestion and rejecting some writes? or is the r bulk api of high level rest client taking care of writing the documents using retry logic? can we increase the size of the write thread pool

[DavidTurner]

I suggest opening a new topic about that - this question is nothing to do with the number of master nodes as the title suggests.

[warkolm (Mark Walkom)]

I would suggest that your cluster is too large, and that you need to look at splitting it into small clusters and then use CCS to query things. It'll make management easier.