We run an ES version 5.4.1 cluster. It takes in between 1-2TB of network session data (netflow processed by BRO devices) and a few other logs. We have hot nodes and warm nodes in the cluster and all works fine except we run an aggregations against a high volume log (bro-conn logs). This log contains all network session data and it is the highest volume log we have. When an aggregation job is run against this data even for a short time period (4-5) mins ES nodes crashes with java heap OutOfMemory error. I put in pretty low values for circuit breaker settings (see below), but didn't help
"indices" : {
"breaker" : {
"fielddata" : {
"limit" : "20%"
},
"request" : {
"limit" : "20%"
},
"total" : {
"limit" : "30%"
}
},
Any ideas?