Whole elasticsearh cluster become unresponsive if only one node is saturating

[shahzadkhan (shahzad)]

Hello All,
we have elasticsearch cluster with 12 nodes and all the nodes are configured as master/data.
these days we encountring an issue that all the queries are automatically forwarded to a only one node and the thread pool on that node get saturated and all other 11 nodes become un responsive.
I have some queries regarding this issue:

1. why all other nodes get un responsive and cluster get down even only single node is saturating.
2. Is there a way to understand that why most of the queries are forwarded to a single and then the thread pool on that node gets saturated.
3. Is it fine to have the configuration that all the nodes of the cluster work as master and data node.

Thank you.

[Christian_Dahlqvist]

@shahzadkhan

why all other nodes get un responsive and cluster get down even only single node is saturating.

Which version of Elasticsearch are you using? What is the specification of the nodes?

@shahzadkhan

Is there a way to understand that why most of the queries are forwarded to a single and then the thread pool on that node gets saturated.

Where are the queries originating from?

@shahzadkhan

Is it fine to have the configuration that all the nodes of the cluster work as master and data node.

When you reach a cluster size of over 10 nodes it is common to set up 3 dedicated master nodes and have the data nodes no longer be master eligible.

[shahzadkhan (shahzad)]

Hello Christian,
The elasticSearch is running on the kubernetes cluster with 6 worker nodes each with 48 cpu e 378GB ram. Each kubernetes worker node host 2 pod/host elascticsaerch each with16 cpu e 116Gb ram. So in total we have 12 elascticseach nodes each with 16 cpu e 116Gb ram. The elasticsearch version is 7.8.0 and the queries originates from our application pod's hosted on the same kubernetes cluster. Let me know if you need any other information , thank you.