Hi,
I'm getting the error below occasionally, and I'm trying to find the cause of it so I can try to resolve it.
circuit_breaking_exception: [parent] Data too large, data for [<http_request>] would be [32620387928/30.3gb], which is larger than the limit of [31621696716/29.4gb], real usage: [32620387928/30.3gb], new bytes reserved: [0/0b], usages [eql_sequence=0/0b, fielddata=24272253421/22.6gb, request=0.0b, inflight_requests=0/0b, model_inference=0.0b]
I'm running ES v8.8.0, with multiple instances of filebeat writing to a single index, at the indexing rate of about 160K/s. The ES cluster is on Kubernetes, with 3 master ES nodes and 40 data nodes (1.3TB of memory (usually at 50-60% usage), 126TB/600TB of SSD space free). Once I hit the error message above, I won't be able to access the Kibana/ES anymore, and would have to restart the ES nodes.
One strange thing I just happened to notice was when I was got the error, the indexing rate of the index currently being written to got up to an impossibly high number, like 9-digit (on the Stack Monitoring --> Indices page, time window: last 2 min). Then it drops to 2+M, which is still unusually high since I'm expecting at most ~160K/s. Not sure if this has anything to do with the error.
Is there a way to find out what is causing this issue, such as some complex query running, or the ingest rate is too high (I have gotten this error at a lower indexing rate of ~100K/s too), etc?
So far, I've noticed that the real usage
is not that much higher than the limit
. Would increasing the limit
from 29.4gb
to, say, 32gb
be possible and feasible?
Thank you.