Elasticsearch shows Circuit Breaking Exception

Elastic Stack Elasticsearch
#1

We have Elasticsearch deployed in a Kubernetes cluster as a 3 node (all 3 are Master and Data).
Recently we are facing issue with Elasticsearch Circuit breaking excpetion and only 2 nodes are reporting to the Cluster. And the Cluster state is always RED.

Cirucit Breaking Exception
Cirucit Breaking Exception1435×365 13.1 KB

The output for allocation/explain returns:
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<transport_request>] would be [6362457143/5.9gb], which is larger than the limit of [5988548608/5.5gb]",
"bytes_wanted" : 6362457143,
"bytes_limit" : 5988548608
}
],
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<transport_request>] would be [6362457143/5.9gb], which is larger than the limit of [5988548608/5.5gb]",
"bytes_wanted" : 6362457143,
"bytes_limit" : 5988548608
},
"status" : 503
}

Can we set the Parent circuit breaker limit value to the required size via ES API?

Regards,
Pavan

#2

The circuit breaker is there to prevent nodes from crashing, so I would recommend first looking into what is causing the high heap usage. What is the full output of the cluster stats API? Do you have any non-standard settings that could impact heap usage?

#3

Hi

You may implement either of below

  1. Increase your Server RAM to have more HEAP_SIZE

  2. You need to adjust jvm.option file below parameters and check what values are optimal for your environment.

10-:-XX:G1ReservePercent=25
10-:-XX:InitiatingHeapOccupancyPercent=30

Thanks
Tushar Nemade

#4

We do not recommend altering these.

1 Like
#5

Okay.

How to fix this Circuit Breaker issue , if it arrises suddenly after few days of running Elasticsearch v7.x as data to be pushed to ES Index has not increased.

#6

Hello Mark,

I just modified the transient indices.breaker.total.limit to the required memory so that the cluster can spin up.
Once the cluster state turned Green, I have reverted the circuit breaker and restarted my ES nodes.
Could you help me to avoid this from happening again? Or if it happens again, is there any document that can help me where to look for the error, configurations to consider to troubleshoot this issue faster?

I probably should have mentioned this with the OP, I am using Elasticsearch 6.3.0.

Regards,
Pavan