I'm wondering what is the effect on a cluster if the master node has very high load (eg - 10-20). The node isn't down, but is slow - what effect will this have on the cluster? This is assuming it's not a data node.
It could potentially affect the stability of the cluster. If that node becomes unresponsive (long GCs, etc), the other dedicated masters may assume it has died and start the election process. If the masters are constantly swapping roles amongst themselves, that can make the cluster much less stable and appear flaky.
Even if the master isn't pausing long enough to lose it's role, the high load could be slowing down important duties like changing the cluster state when new indices are added, new nodes joining/leaving, etc. So it could affect the "realtime-ness" of certain operations, since the tasks will pend until the master can get to them.
Generally speaking, you want your master nodes to be doing absolutely nothing other than dealing with ES tasks, to maximize stability.
just what I needed - thanks!