# GC: collection\_count questions **URL:** \ **Category:** Elasticsearch\ **Created:** [September 10, 2015, 2:55pm UTC](https://discuss.elastic.co/t/gc-collection-count-questions/29041 "2015-09-10T14:55:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![genesis](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@genesis](https://discuss.elastic.co/u/genesis)\ **Post date:** [September 10, 2015, 2:55pm UTC](https://discuss.elastic.co/t/gc-collection-count-questions/29041/1 "2015-09-10T14:55:03Z")
We have been having some issues with our production cluster composed on 7 nodes. Right now the cluster is green but I see something that is making feel comfortable about its state. "version" : { "number" : "1.7.1", java full version "1.8.0\_51-b16" I ran the node stats command and I noticed the "gc" collectors "old" are too high on all nodes. According to what I read online " The old generation collection count should remain small, and have a small collection\_time\_in\_millis" ``` "gc" : { "collectors" : { "young" : { "collection_count" : 14830, "collection_time_in_millis" : 993189 }, "old" : { "collection_count" : 2857, "collection_time_in_millis" : 1082151 } } ``` ## -- "gc" : { "collectors" : { "young" : { "collection\_count" : 14127, "collection\_time\_in\_millis" : 947026 }, "old" : { "collection\_count" : 2809, "collection\_time\_in\_millis" : 1814255 } } ``` "gc" : { "collectors" : { "young" : { "collection_count" : 27328, "collection_time_in_millis" : 2547057 }, "old" : { "collection_count" : 2493, "collection_time_in_millis" : 284310 } } ``` ## -- "gc" : { "collectors" : { "young" : { "collection\_count" : 15054, "collection\_time\_in\_millis" : 894937 }, "old" : { "collection\_count" : 2829, "collection\_time\_in\_millis" : 511283 } } ``` "gc" : { "collectors" : { "young" : { "collection_count" : 13068, "collection_time_in_millis" : 1075700 }, "old" : { "collection_count" : 2636, "collection_time_in_millis" : 378742 } } ``` ## -- "gc" : { "collectors" : { "young" : { "collection\_count" : 13100, "collection\_time\_in\_millis" : 1208937 }, "old" : { "collection\_count" : 2592, "collection\_time\_in\_millis" : 714342 } } ``` "gc" : { "collectors" : { "young" : { "collection_count" : 22154, "collection_time_in_millis" : 1379513 }, "old" : { "collection_count" : 2701, "collection_time_in_millis" : 281411 } } ``` The server uptime is as follow: Node1: 1 day Node2: 28 days Node3: 28 days Node4: 28 days Node5: 28 days Node6: 20 days Node7 1 day ## I also noticed the heap\_used\_percent is most of the time above 75% "jvm" : { "timestamp" : 1441893597402, "uptime\_in\_millis" : 145190322, "mem" : { "heap\_used\_in\_bytes" : 19953288096, "heap\_used\_percent" : 84, "heap\_committed\_in\_bytes" : 23587454976, "heap\_max\_in\_bytes" : 23587454976, "non\_heap\_used\_in\_bytes" : 129210576, "non\_heap\_committed\_in\_bytes" : 132513792, "pools" : { ``` "jvm" : { "timestamp" : 1441893597241, "uptime_in_millis" : 145200995, "mem" : { "heap_used_in_bytes" : 20344159872, "heap_used_percent" : 86, "heap_committed_in_bytes" : 23587454976, "heap_max_in_bytes" : 23587454976, "non_heap_used_in_bytes" : 138011320, "non_heap_committed_in_bytes" : 140562432, "pools" : { "jvm" : { "timestamp" : 1441893596704, "uptime_in_millis" : 143036366, "mem" : { "heap_used_in_bytes" : 20152739200, "heap_used_percent" : 81, "heap_committed_in_bytes" : 24661196800, "heap_max_in_bytes" : 24661196800, "non_heap_used_in_bytes" : 125541880, "non_heap_committed_in_bytes" : 128040960, "pools" : { ``` This is my elasticsearch.yml setting path.data: /var/data/elasticsearch cluster.name: GM-RTD node.master: true node.name: ElasticSearch-1 http.cors.enabled: true plugin.mandatory: cloud-aws bootstrap.mlockall: true cloud.aws.access\_key: XXXXXX cloud.aws.secret\_key: XXXXXXX discovery.type: ec2 discovery.zen.ping.multicast.enabled: false discovery.ec2.groups: GM-VPC discovery.zen.minimum\_master\_nodes: 1 gateway.recover\_after\_nodes: 1 gateway.recover\_after\_time: 5m gateway.expected\_nodes: 2 The heap size is set to: ES\_HEAP\_SIZE=22g sudo sysctl -a | grep vm.max\_map\_count vm.max\_map\_count = 262144 process" : { "refresh\_interval\_in\_millis" : 1000, "id" : 32383, "max\_file\_descriptors" : 65535, "mlockall" : true } ---
**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\ **Post date:** [September 10, 2015, 3:35pm UTC](https://discuss.elastic.co/t/gc-collection-count-questions/29041/2 "2015-09-10T15:35:01Z")
When the percent of heap used is between 70% and 80%, that is a good signal that it's time to either add memory or look into what's using memory. Might very well be that it will also explain the reason behind the many old collection counts. Are you using aggregations, sorting or scripting? Those are the features that make heavy use of field\_data cache, the main responsible for high heap usage. Also which version of elasticsearch are you on? ---
**Author:** ![genesis](https://avatars.discourse-cdn.com/v4/letter/g/96bed5/32.png) [@genesis](https://discuss.elastic.co/u/genesis)\ **Post date:** [September 10, 2015, 6:59pm UTC](https://discuss.elastic.co/t/gc-collection-count-questions/29041/3 "2015-09-10T18:59:00Z")
Thanks for getting back to us, here are the info you requested: version" : { "number" : "1.7.1", "build\_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19", "build\_timestamp" : "2015-07-29T09:54:16Z", "build\_snapshot" : false, "lucene\_version" : "4.10.4" Yes, our Analytic team does lot of aggreations and scripting. These are the field data per node "fielddata" : { "limit\_size\_in\_bytes" : 14152472985, "limit\_size" : "13.1gb", "estimated\_size\_in\_bytes" : 0, "estimated\_size" : "0b", "overhead" : 1.03, "tripped" : 0 ``` "fielddata" : { "memory_size_in_bytes" : 4739124, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ## -- "fielddata" : { "limit\_size\_in\_bytes" : 14152472985, "limit\_size" : "13.1gb", "estimated\_size\_in\_bytes" : 4739124, "estimated\_size" : "4.5mb", "overhead" : 1.03, "tripped" : 0 }, "request" : { "limit\_size\_in\_bytes" : 9434981990, "limit\_size" : "8.7gb", ``` "fielddata" : { "memory_size_in_bytes" : 106943440, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ## -- "fielddata" : { "limit\_size\_in\_bytes" : 14152472985, "limit\_size" : "13.1gb", "estimated\_size\_in\_bytes" : 106943440, "estimated\_size" : "101.9mb", "overhead" : 1.03, "tripped" : 0 }, "request" : { "limit\_size\_in\_bytes" : 9434981990, "limit\_size" : "8.7gb", ``` "fielddata" : { "memory_size_in_bytes" : 53842652, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ## -- "fielddata" : { "limit\_size\_in\_bytes" : 14152472985, "limit\_size" : "13.1gb", "estimated\_size\_in\_bytes" : 53842652, "estimated\_size" : "51.3mb", "overhead" : 1.03, "tripped" : 0 }, "request" : { "limit\_size\_in\_bytes" : 9434981990, "limit\_size" : "8.7gb", ``` "fielddata" : { "memory_size_in_bytes" : 0, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ## -- "fielddata" : { "limit\_size\_in\_bytes" : 14796718080, "limit\_size" : "13.7gb", "estimated\_size\_in\_bytes" : 0, "estimated\_size" : "0b", "overhead" : 1.03, "tripped" : 0 }, "request" : { "limit\_size\_in\_bytes" : 9864478720, "limit\_size" : "9.1gb", ``` "fielddata" : { "memory_size_in_bytes" : 3992540, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ## -- "fielddata" : { "limit\_size\_in\_bytes" : 14152472985, "limit\_size" : "13.1gb", "estimated\_size\_in\_bytes" : 3992540, "estimated\_size" : "3.8mb", "overhead" : 1.03, "tripped" : 0 }, "request" : { "limit\_size\_in\_bytes" : 9434981990, "limit\_size" : "8.7gb", ``` "fielddata" : { "memory_size_in_bytes" : 71535588, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ## -- "fielddata" : { "limit\_size\_in\_bytes" : 14152472985, "limit\_size" : "13.1gb", "estimated\_size\_in\_bytes" : 71535588, "estimated\_size" : "68.2mb", "overhead" : 1.03, "tripped" : 0 }, "request" : { "limit\_size\_in\_bytes" : 9434981990, "limit\_size" : "8.7gb", ``` "fielddata" : { "memory_size_in_bytes" : 52855724, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 11, 2015, 5:35am UTC](https://discuss.elastic.co/t/gc-collection-count-questions/29041/4 "2015-09-11T05:35:31Z")
Then as mentioned you probably need to scale to cope with your load. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:50pm UTC](https://discuss.elastic.co/t/gc-collection-count-questions/29041/5 "2017-07-05T23:50:58Z")