We are having the following cluster setup
1 master node
2 client nodes
2 data nodes
Master node:
8 Gb RAM
Heap: -Xms2g -Xmx2g
Each Client nodes:
16 Gb RAM
Heap: -Xms8g -Xmx8g
Each Data node
16 Gb RAM
Heap: -Xms8g -Xmx8g
Elasticsearch version: 5.2.0
we are pushing data from mysql to elasticsearch through logstash. logstash is configured to use one es-client (lets call it es-client-2). Once this setup was ready it worked for a nearly a week before it started throwing the following issue:
[INFO ][o.e.d.z.ZenDiscovery ] [es-client-2] failed to send join request to master [{es-master}{zQzKnjSETXuavCd0OXVv9A}{bUMt-luHQpOfHwAFo1DmMg}{es-master}{10.20.10.40:9300}], reason [RemoteTransportException[[es-master][10.20.10.40:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[es-client-2][10.20.10.81:9300][internal:discovery/zen/join/validate]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be larger than limit of [5988548608/5.5gb]]; ]
Once i restarted this client, it joined the cluster and started working as normal.
I am trying to find the root cause of this issue. Why would it reach a limit of 5.5 gb. At the same time if i try with the other es client, it works.
Can you throw some light on this issue?