I have a Elasticsearch cluster with around more than 100 data nodes. around 30 hot nodes and the rest warm nodes.
I am coming across an issue where I just try to load data with the index pattern say logs* in kibana discover,
Then I see a shard error saying circuit breaking exception for a shard that is on a specific node.
say data-node-a.
I resolved it just by moving that shard manually from that specific node.
This has been occuring multiple times, as I move the shard away from the same node dat-node-a,
the issue gets resolved.
Now,
To troubleshoot this or to understand why the shards on that node are giving me trouble,
what all I would need to check on that node and how do I identify the cause.
please point me to all the checks I would need to do on that. I might get to do all those tests and identify the root cause next time it occurs.
The error is:
circuit_breaking_exception:
data too large so and so (memory higher than it supposed to have ) [parent] breaker.