Finding out cause of circuit breaking exception (Data too large)

Elastic Stack Elasticsearch
3

Hi @carly.richmond ,

Thanks for replying.

Out of the 102 fields I have, 9 are "text" fields.

I'm not sure how to interpret the output of GET _nodes/stats/breaker, and I can't copy and paste the entire output as my ES is on another (offline) machine. Perhaps you can let me know which fields I should be looking at?

Here is a portion of the output (of one of the nodes):

"attributes": {
    ...
    "ml.allocated_processors_double": "12.0",
    "data": "warm",
    "ml.allocated_processors": "12",
    "ml.machine_memory": "68719476736",
    "ml.max_jvm_size": "33285996544"
},
"breakers": {
    "eql_sequence": {
        "limit_size_in_bytes": 16642998272,
        "limit_size": "15.5gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1,
        "tripped": 0
    },
    "fielddata": {
        "limit_size_in_bytes": 13314398617,
        "limit_size": "12.3gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1.03,
        "tripped": 0
    },
    "request": {
        "limit_size_in_bytes": 19971597926,
        "limit_size": "18.5gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1,
        "tripped": 0
    },
    "inflight_requests": {
        "limit_size_in_bytes": 33285996544,
        "limit_size": "31gb",
        "estimated_size_in_bytes": 7412747,
        "estimated_size": "7mb",
        "overhead": 2,
        "tripped": 0
    },
    "model_inference": {
        "limit_size_in_bytes": 16642998272,
        "limit_size": "15.5gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1,
        "tripped": 0
    },
    "parent": {
        "limit_size_in_bytes": 31621696716,
        "limit_size": "29.4gb",
        "estimated_size_in_bytes": 16919285712,
        "estimated_size": "15.7gb",
        "overhead": 1,
        "tripped": 0
    }
}

All tripped values of all nodes are 0.

One observation I made was when the circuit breaking exception happened yesterday again, the JVM heap memory usage was at least 85%.