Circuit_breaking_exception

[karthik_s1 (Karthik S)]

We are having the following cluster setup

1 master node
2 client nodes
2 data nodes

Master node:
8 Gb RAM
Heap: -Xms2g -Xmx2g

Each Client nodes:
16 Gb RAM
Heap: -Xms8g -Xmx8g

Each Data node
16 Gb RAM
Heap: -Xms8g -Xmx8g

Elasticsearch version: 5.2.0

we are pushing data from mysql to elasticsearch through logstash. logstash is configured to use one es-client (lets call it es-client-2). Once this setup was ready it worked for a nearly a week before it started throwing the following issue:

[INFO ][o.e.d.z.ZenDiscovery ] [es-client-2] failed to send join request to master [{es-master}{zQzKnjSETXuavCd0OXVv9A}{bUMt-luHQpOfHwAFo1DmMg}{es-master}{10.20.10.40:9300}], reason [RemoteTransportException[[es-master][10.20.10.40:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[es-client-2][10.20.10.81:9300][internal:discovery/zen/join/validate]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be larger than limit of [5988548608/5.5gb]]; ]

Once i restarted this client, it joined the cluster and started working as normal.

I am trying to find the root cause of this issue. Why would it reach a limit of 5.5 gb. At the same time if i try with the other es client, it works.

Can you throw some light on this issue?

[jasontedor]

Versions less than 5.2.2 had a leak in circuit breaker accounting that could occur with certain clients. Please upgrade to at least 5.2.2 to see if your problem is addressed by this bug fix: #23310.

[karthik_s1 (Karthik S)]

Thanks @jasontedor, we will try it out.

[karthik_s1 (Karthik S)]

@jasontedor i have one more question.

Till we faced this issue for the first time, all our inserts and upserts were working fine. once this issue started appearing we are missing some upserts. Does this issue affect upserts in any manner? We are having this doubt, as there had been no change in the application/ elasticsearch cluster configuration during that period.

[karthik_s1 (Karthik S)]

@jasontedor May i know your opinion on this?

[jasontedor]

The issue I referenced should not have any impact on upserts specifically, only that requests in general would be rejected once the circuit breaker has tripped.