# Elasticsearch Heap size growing with time and lot of GC, eventually pulling the cluster down **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2015, 12:23am UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098 "2015-11-08T00:23:31Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ramindar](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@ramindar](https://discuss.elastic.co/u/ramindar)\ **Post date:** [November 8, 2015, 12:23am UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/1 "2015-11-08T00:23:31Z")
I have a ES cluster with 16 nodes in Azure 3 Master (A2 machine - 3.5GB mem) 3 Search (A2 machine - 3.5GB mem) 10 Data Nodes (A3 machine - 7 GB mem) In total 6 indexes Index Doc Size Shards Replicas Status index-1 64,068 249.6MB 2 2 open index-2 93 2.6MB 2 2 open index-3 24,502 24.4MB 2 2 open index-4 12,697 4.3MB 2 2 open index-5 17,556 7.0MB 2 2 open index-6 347,483 92.8MB 2 3 open Elastic Search Version - 1.7.2 OS: Windows Server 2012 R2 (6.3) Java Version: 1.8.0\_60 JVM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23, mixed mode) Java Vendor: Oracle Corporation elasticsearch.yml bootstrap.mlockall: true indices.breaker.total.limit: 70% indices.fielddata.cache.size: 70% Logs: [2015-11-07 20:30:27,688][WARN][monitor.jvm] [ES-PROD-12-data] [gc][old][205968][5993] duration [22.4s], collections [1]/[23.1s], total [22.4s]/[44.8m], memory [3.4gb]-\>[3.1gb]/[3.4gb], all\_pools {[young] [252.1mb]-\>[2.2kb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.1gb]-\>[3.1gb]/[3.1gb]} [2015-11-07 20:31:02,485][WARN][monitor.jvm] [ES-PROD-12-data] [gc][old][205969][5994] duration [33.7s], collections [1]/[34.7s], total [33.7s]/[45.3m], memory [3.1gb]-\>[3.1gb]/[3.4gb], all\_pools {[young] [2.2kb]-\>[116.2kb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.1gb]-\>[3.1gb]/[3.1gb]} [2015-11-07 20:31:26,313][WARN][monitor.jvm] [ES-PROD-12-data] [gc][old][205970][5995] duration [22.6s], collections [1]/[23.8s], total [22.6s]/[45.7m], memory [3.1gb]-\>[3.1gb]/[3.4gb], all\_pools {[young] [116.2kb]-\>[6.2mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.1gb]-\>[3.1gb]/[3.1gb]} [2015-11-07 20:32:01,125][WARN][monitor.jvm] [ES-PROD-12-data] [gc][old][205972][5996] duration [33.3s], collections [1]/[33.7s], total [33.3s]/[46.3m], memory [3.4gb]-\>[3.1gb]/[3.4gb], all\_pools {[young] [266.2mb]-\>[5.5mb]/[266.2mb]}{[survivor] [8.5mb]-\>[0b]/[33.2mb]}{[old] [3.1gb]-\>[3.1gb]/[3.1gb]} [2015-11-07 20:32:24,718][WARN][monitor.jvm] [ES-PROD-12-data] [gc][old][205974][5997] duration [22.2s], collections [1]/[22.5s], total [22.2s]/[46.6m], memory [3.4gb]-\>[3.1gb]/[3.4gb], all\_pools {[young] [266.2mb]-\>[9.4mb]/[266.2mb]}{[survivor] [15.3mb]-\>[0b]/[33.2mb]}{[old] [3.1gb]-\>[3.1gb]/[3.1gb]} [2015-11-07 20:32:58,937][WARN][monitor.jvm] [ES-PROD-12-data] [gc][old][205976][5998] duration [32.5s], collections [1]/[33.2s], total [32.5s]/[47.2m], memory [3.4gb]-\>[3.1gb]/[3.4gb], all\_pools {[young] [254.3mb]-\>[17.4mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [3.1gb]-\>[3.1gb]/[3.1gb]} GC kicks in every minute and takes around have a minute, eventually the queries queue up and we start getting the exceptions in the Search Nodes org.elasticsearch.transport.RemoteTransportException: [ES-PROD-12-data][inet[/10.1.0.7:9300]][indices:data/read/search[phase/query]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler@5b5e7b85 at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:79) at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:224) ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [November 8, 2015, 1:15am UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/2 "2015-11-08T01:15:24Z")
Hi Raminder, I think indices.fielddata.cache.size: 70% is too high. Try reducing that to a very low value and instead of that set doc\_values on all you attributes. This should significantly reduce the load on the heap. The disk utilization will increase though. Regards, Srinath ---
**Author:** ![ramindar](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@ramindar](https://discuss.elastic.co/u/ramindar)\ **Post date:** [November 10, 2015, 6:53pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/3 "2015-11-10T18:53:38Z")
Thanks Srinath. I reduced the indices.fielddata.cache.size: to 10% now, and the heap seems to happy for few days. Post that everyday the Heap size seems to be growing again. even thought the data remains the same. Not sure if i regularly run optimize over the indexes, would that help ? ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [November 11, 2015, 12:12am UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/4 "2015-11-11T00:12:52Z")
Reducing indices.fielddata.cache.size: to 10% was not the fix. Setting doc\_values on properties in the index mappings was the solution. With that there will be a reduced load on memory. Did you set doc\_values and try? ---
**Author:** ![ramindar](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@ramindar](https://discuss.elastic.co/u/ramindar)\ **Post date:** [November 12, 2015, 7:41am UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/5 "2015-11-12T07:41:50Z")
Hi Srinath, I tried with the doc\_values as well. but the problem still persists. After few hours the **old gen memory** grows to its limit and GC kicks in and same issue starts again. also there is another error org.elasticsearch.transport.RemoteTransportException: [ES-PROD-9-data][inet[/10.1.0.18:9300]][indices:data/read/search[phase/fetch/id]] Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [1068983] at org.elasticsearch.search.SearchService.findContext(SearchService.java:537) at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:507) at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868) at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) I took the heap dump and observed that most of the memory, 25% is used by **ConcurrentHahMap** and 25% is used by **string**. Not sure what is happening ☹ ---
**Author:** ![ramindar](https://avatars.discourse-cdn.com/v4/letter/r/8edcca/32.png) [@ramindar](https://discuss.elastic.co/u/ramindar)\ **Post date:** [November 20, 2015, 4:13am UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/6 "2015-11-20T04:13:33Z")
I thought memory is the problem, so i tried to increase the VM size to A6 having 28gb memory, but still i got the same issue. **Search Thread count goes to 1000 and then huge set of exceptions** org.elasticsearch.transport.RemoteTransportException: [ES-PROD-12-data][inet[/10.1.0.7:9300]][indices:data/read/search[phase/query]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on **any help would be greatly appreciated.** ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:37pm UTC](https://discuss.elastic.co/t/elasticsearch-heap-size-growing-with-time-and-lot-of-gc-eventually-pulling-the-cluster-down/34098/7 "2017-07-05T23:37:04Z")