# Elasticsearch operational issue due to garbage collector **URL:** \ **Category:** Elasticsearch\ **Created:** [September 28, 2017, 5:38pm UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158 "2017-09-28T17:38:03Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\ **Post date:** [September 28, 2017, 5:38pm UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/1 "2017-09-28T17:38:03Z")
running a 4 node elasticsearch cluster. which has one client node and 3 data nodes. data nodes are also configured as master eligible node. elasticsearch version is 2.3.5 lucene version is 5.5.0 JAVA is openjdk version 1.8.0\_141 below are the JAVA\_OPTS values with which elasticsearch is running. -Xms10g -Xmx10g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC when ever GC is triggered and at that time data nodes leaves the cluster. below are the GC logs: [2017-09-28 11:52:52,186][WARN][monitor.jvm] [10.0.1.85] [gc][old][504680][614] duration [39.8m], collections [84]/[40.9m], total [39.8m]/[5h], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.4mb]/[532.5mb]}{[survivor] [66.4mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2017-09-28 11:52:53,186][WARN][monitor.jvm] [10.0.1.85] [gc][old][504681][615] duration [29.5s], collections [1]/[30.5s], total [29.5s]/[5h], memory [9.9gb]-\>[6.1gb]/[9.9gb], all\_pools {[young] [532.4mb]-\>[47.1mb]/[532.5mb]}{[survivor] [66.4mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[6gb]/[9.3gb]} during this time cluster doesn't process the queries. and queries fails with [2017-09-28 12:34:49,232][DEBUG][action.admin.cluster.health] [10.0.1.85] no known master node, scheduling a retry [2017-09-28 12:35:19,233][DEBUG][action.admin.cluster.health] [10.0.1.85] timed out while retrying [cluster:monitor/health] after failure (timeout [30s]) [2017-09-28 12:35:19,234][WARN][rest.suppressed] path: /\_cluster/health, params: {} MasterNotDiscoveredException[null] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226) at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236) at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) and data nodes are unable to discover the master nodes, below are the logs [2017-09-28 12:33:48,614][WARN][discovery.zen.ping.unicast] [10.0.1.85] failed to send ping to [{#zen\_unicast\_3#}{10.0.1.89}{10.0.1.89:9300}] ReceiveTimeoutTransportException[[][10.0.1.89:9300][internal:discovery/zen/unicast] request\_id [648779] timed out after [3750ms]] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) really require suggestions on this, how we can optimise it. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 28, 2017, 5:49pm UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/2 "2017-09-28T17:49:08Z")
Can you provide the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/cluster-stats.html#cluster-stats)? ---
**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\ **Post date:** [September 28, 2017, 5:53pm UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/3 "2017-09-28T17:53:46Z")
{ "timestamp" : 1506621190822, "cluster\_name" : "pediatric", "status" : "green", "indices" : { "count" : 17, "shards" : { "total" : 170, "primaries" : 85, "replication" : 1.0, "index" : { "shards" : { "min" : 10, "max" : 10, "avg" : 10.0 }, "primaries" : { "min" : 5, "max" : 5, "avg" : 5.0 }, "replication" : { "min" : 1.0, "max" : 1.0, "avg" : 1.0 } } }, "docs" : { "count" : 15497625, "deleted" : 0 }, "store" : { "size" : "25.2gb", "size\_in\_bytes" : 27117215234, "throttle\_time" : "0s", "throttle\_time\_in\_millis" : 0 }, "fielddata" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "query\_cache" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "total\_count" : 0, "hit\_count" : 0, "miss\_count" : 0, "cache\_size" : 0, "cache\_count" : 0, "evictions" : 0 }, "completion" : { "size" : "0b", "size\_in\_bytes" : 0 }, "segments" : { "count" : 901, "memory" : "133.7mb", "memory\_in\_bytes" : 140246775, "terms\_memory" : "117.4mb", "terms\_memory\_in\_bytes" : 123143587, "stored\_fields\_memory" : "11.6mb", "stored\_fields\_memory\_in\_bytes" : 12214120, "term\_vectors\_memory" : "0b", "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory" : "4.5mb", "norms\_memory\_in\_bytes" : 4791360, "doc\_values\_memory" : "95.4kb", "doc\_values\_memory\_in\_bytes" : 97708, "index\_writer\_memory" : "0b", "index\_writer\_memory\_in\_bytes" : 0, "index\_writer\_max\_memory" : "83mb", "index\_writer\_max\_memory\_in\_bytes" : 87040000, "version\_map\_memory" : "0b", "version\_map\_memory\_in\_bytes" : 0, "fixed\_bit\_set" : "0b", "fixed\_bit\_set\_memory\_in\_bytes" : 0 }, "percolate" : { "total" : 0, "time" : "0s", "time\_in\_millis" : 0, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 0 } }, "nodes" : { "count" : { "total" : 4, "master\_only" : 0, "data\_only" : 0, "master\_data" : 3, "client" : 0 }, "versions" : ["2.3.5"], "os" : { "available\_processors" : 26, "allocated\_processors" : 26, "mem" : { "total" : "30.4gb", "total\_in\_bytes" : 32718340096 }, "names" : [ { "name" : "Linux", "count" : 4 } ] }, "process" : { "cpu" : { "percent" : 0 }, "open\_file\_descriptors" : { "min" : 191, "max" : 845, "avg" : 657 } }, "jvm" : { "max\_uptime" : "5.1h", "max\_uptime\_in\_millis" : 18645205, "versions" : [ { "version" : "1.8.0\_141", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "25.141-b16", "vm\_vendor" : "Oracle Corporation", "count" : 4 } ], "mem" : { "heap\_used" : "1.2gb", "heap\_used\_in\_bytes" : 1315919824, "heap\_max" : "32.7gb", "heap\_max\_in\_bytes" : 35206856704 }, "threads" : 241 }, "fs" : { "total" : "1.4tb", "total\_in\_bytes" : 1584931799040, "free" : "572.3gb", "free\_in\_bytes" : 614515998720, "available" : "497.2gb", "available\_in\_bytes" : 533935030272 }, "plugins" : [ { "name" : "head", "version" : "master", "description" : "head - A web front end for an elastic search cluster", "url" : "/\_plugin/head/", "jvm" : false, "site" : true }, { "name" : "sql", "version" : "2.3.5.0", "description" : "Query elasticsearch using SQL", "url" : "/\_plugin/sql/", "jvm" : true, "classname" : "org.elasticsearch.plugin.nlpcn.SqlPlug", "isolated" : true, "site" : true } ] } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 28, 2017, 6:07pm UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/4 "2017-09-28T18:07:22Z")
Do you have any graphs of how heap usage varies over time? It seems like something uses up a lot of heap, leading to long GC. Heap usage however currently seem to be low and all seems fine as far as I can tell. I have however never used the SQL plugin you have installed so have no idea how this affects heap usage. ---
**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\ **Post date:** [September 28, 2017, 6:20pm UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/5 "2017-09-28T18:20:14Z")
generally, we do not face any issue during normal operation. we are using elastic node with Hadoop clusters to store the output of jobs which were executed on 1 0r 2 TB of data. and also we do not face this issue on daily basis. it comes once in 7 or 8 days. I suspect issue due to JAVA opts arguments for garbage collector. **-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC** openJDK 1.8 is used. ---
**Author:** ![chitender](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chitender/32/22621_2.png) [@chitender](https://discuss.elastic.co/u/chitender)\ **Post date:** [September 29, 2017, 7:04am UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/6 "2017-09-29T07:04:45Z")
I am preparing the bash script to gather the heap usage of all nodes. let me know what are parameters I should consider for analysing the heap usage. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 29, 2017, 7:06am UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/7 "2017-09-29T07:06:33Z")
It would be interesting to see how it develops over time, so I would recommend installing monitoring. Based on the point in time stats you provided it looks fine. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 27, 2017, 7:06am UTC](https://discuss.elastic.co/t/elasticsearch-operational-issue-due-to-garbage-collector/102158/8 "2017-10-27T07:06:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.