# GC Pauses and OOM errors when indexing into a 800 GB cluster
**URL:** https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209
**Category:** Elasticsearch
**Created:** [July 9, 2015, 6:36am UTC](https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209 "2015-07-09T06:36:24Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@naryad](https://discuss.elastic.co/u/naryad)
#### Post date: [July 9, 2015, 6:36am UTC](https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209/1 "2015-07-09T06:36:25Z")
I have a 5 node ES Cluster with 32 Gig RAM on each node. I assign 20GB to ES process. These are relevant fields in my yml.
discovery.type: ec2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.timeout: 60s (changed this as 10 s was not enough)
discovery.zen.minimum\_master\_nodes: 3
script.disable\_dynamic: true
bootstrap.mlockall: true
indices.fielddata.cache.size: 50%
indices.breaker.fielddata.limit: 60%
indices.breaker.request.limit: 40%
indices.breaker.total.limit: 70%
Elasticsearch Version: 1.3.1
I index anywhere between 500-1000 documents per minute (which are structured more like tweets and social network data). I have 406 Million documents in my cluster (replica excluded) and 800 GB of data (replica included).
Recently I observed heap continuously increasing and in the end, GC pause of OOM takes nodes down. I figured that this is more of a problem with indexing than querying as the field data cache and filter cache never exceeds 3 GB combined.
This is the current cluster health
{
"cluster\_name": "name\_of\_cluster",
"status": "yellow",
"timed\_out": false,
"number\_of\_nodes": 5,
"number\_of\_data\_nodes": 5,
"active\_primary\_shards": 15,
"active\_shards": 26,
"relocating\_shards": 0,
"initializing\_shards": 4,
"unassigned\_shards": 0
}
I would like to know where I need to do improvements. Should I increase the RAM to 64 G per node and such options. I am also considering using doc\_values and upgrading ES to latest version. But I would like to understand the root cause of this behaviour before taking any action.
This is the hot threads output [https://gist.github.com/naryad/abe852c04dbac5e5611a](https://gist.github.com/naryad/abe852c04dbac5e5611a)
This is the output of node stats API [https://gist.github.com/naryad/06ec0e17c0c02e311e80](https://gist.github.com/naryad/06ec0e17c0c02e311e80)
After heap gets filled slowly (old generation objects) and GC happens none of the old generation objects get cleared because of GC. Old generation objects account up to 90% of the 20 GB heap allocated to ES.
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [July 9, 2015, 8:54am UTC](https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209/2 "2015-07-09T08:54:31Z")
Can you update to a more recent ES version?
---
### Author:  [@naryad](https://discuss.elastic.co/u/naryad)
#### Post date: [July 9, 2015, 8:58am UTC](https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209/3 "2015-07-09T08:58:04Z")
I can do that. In fact that is the immediate next thing I am gonna do. Thanks for the reply. Just wanted to know if I am doing something wrong or do I need to update RAM to 64 GB RAM or increase the number of nodes etc. or tune any settings.
---
### Author:  [@naryad](https://discuss.elastic.co/u/naryad)
#### Post date: [July 10, 2015, 6:24am UTC](https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209/4 "2015-07-10T06:24:23Z")
Looks like upgrade has some solid effect. No longer does the heap continuously fill.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:02am UTC](https://discuss.elastic.co/t/gc-pauses-and-oom-errors-when-indexing-into-a-800-gb-cluster/25209/5 "2017-07-06T00:02:23Z")