# OOM on aggregation and lot of time out exceptions
**URL:** https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894
**Category:** Elasticsearch
**Created:** [May 1, 2016, 7:33pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894 "2016-05-01T19:33:55Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 1, 2016, 7:33pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/1 "2016-05-01T19:33:55Z")
Hey All,
I am running three node cluster setup with ES 2.2.0. Each is 32 core 64GB instance and ES has around 17GB of RAM allocated. I have around 350million records. All my three nodes are performing very badly with all possible kind of exception.
Few of them are below
1. `ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [myindex-2016-05-01T12]) within 30s]`
`4. java.util.concurrent.TimeoutException: Failed to acknowledge mapping update within [30s]`
`5. [2016-05-01 12:29:01,806][WARN][transport] [Node2] Received response for a request that has timed out, sent [17789ms] ago, timed out [2788ms] ago, action [cluster:monitor/nodes/stats[n]], node [{Node0}{HQbDpWZ7RcGIOEoKslkR2Q}{17.30.25.25}{17.30.25.25:9300}{master=true}], id [369038]`
And I have below setting in elasticsearch,
# ---------------------------------- Cache Size --------------------------------
indices.fielddata.cache.size: 70%
indices.breaker.fielddata.limit: 75%
# ---------------------------------- Thread pool --------------------------------
threadpool.index.queue\_size: 2000
threadpool.search.queue\_size: 2000
threadpool.bulk.queue\_size: 2000
bootstrap.mlockall: true
indices.store.throttle.max\_bytes\_per\_sec: 100mb
I am aggregating some index for last 24hours records every 10min since new data will be added every min.
Can someone please suggest what's going wrong here. I see OOm bcz of aggregation but how can I avoid this?
Regards,
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 1, 2016, 10:35pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/2 "2016-05-01T22:35:29Z")
> [@Pradeep\_Gowda](#):
>
> indices.fielddata.cache.size: 70%
You are giving 70% of your heap to fielddata, this is not good.
> [@Pradeep\_Gowda](#):
>
> threadpool.index.queue\_size: 2000
> threadpool.search.queue\_size: 2000
> threadpool.bulk.queue\_size: 2000
Neither are these, given that threadpools are held in the JVM memory.
So with the fielddata and over inflated queues, it's no wonder you are OOMing.
I'd suggest you need to look into doc values ASAP. In the short term you either need to reduce your query size or increase the resources available to ES, by either adding more nodes to the cluster or by adding more RAM to them.
---
### Author:  [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 1, 2016, 11:11pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/3 "2016-05-01T23:11:51Z")
What should be the idle fielddata.cache.size? I added this to avoid OOM. There is lot of aggregation happening in my case. How should I give the configuration.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 1, 2016, 11:20pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/4 "2016-05-01T23:20:45Z")
Hitting the filter cache limit won't cause an OOM, it'll actually stop that from happening thanks to the circuit breakers.
---
### Author:  [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 1, 2016, 11:23pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/5 "2016-05-01T23:23:30Z")
Sure, will take a look into circuit breakers. but I'm not understanding why each node is getting timeout. All my nodes are going crazy. A single cat request is taking like a min to respond.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 1, 2016, 11:26pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/6 "2016-05-01T23:26:16Z")
Your cluster is overloaded.
How much data is that 350M docs? How many indices and shards?
---
### Author:  [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 1, 2016, 11:56pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/7 "2016-05-01T23:56:30Z")
Below is the index detail. I do aggregation only on index6, index9 and index10. For aggregation, I query only for last 24 hours.
So the total data which I have now is for a week.
```
green open index1 5 1 10232 0 3.1mb 1.5mb
green open index2 5 1 4132 0 4.9mb 2.4mb
green open index3 5 1 2211427 0 1021.5mb 510.6mb
green open index4 5 1 45930380 142328 10.8gb 5.4gb
green open index5 5 1 3353 0 13.9mb 6.9mb
green open index6 5 1 183713123 0 61.2gb 30.7gb
green open index7 5 1 175 163 321.6kb 158kb
green open index8 5 1 6387812 0 3.6gb 1.8gb
green open index9 5 1 294701932 0 78.6gb 39.1gb
green open index10 5 1 50690374 0 21.2gb 10gb
green open index11 5 1 3046745 0 2gb 1gb
green open index12 5 1 495746 0 1.9gb 993.8mb
green open index13 5 1 10232 0 3mb 1.5mb
green open index14 5 1 140825 15 171mb 85.5mb
green open index15 5 1 326 33 379.4kb 165.9kb
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:54pm UTC](https://discuss.elastic.co/t/oom-on-aggregation-and-lot-of-time-out-exceptions/48894/8 "2017-07-05T22:54:57Z")