# Gc takes a lot of time **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2018, 2:24pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539 "2018-01-08T14:24:47Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![gensmusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gensmusic/32/39173_2.png) [@gensmusic](https://discuss.elastic.co/u/gensmusic)\ **Post date:** [January 8, 2018, 2:24pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/1 "2018-01-08T14:24:47Z")
we use elasticsearch to store log data. Currently we have 16 data nodes and 3 master. Each data node is 24 core cpu, 2.6Tx12 disk, and 32G memory. We use half of the memory for elasticsearch. We use index name like bussness-2018.01.01. When the shards number grow to 27000, the gc begins like ``` [2018-01-08T13:32:13,437][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][1295560] overhead, spent [15.4s] collecting in the last [16.2s] [2018-01-08T13:32:53,044][INFO][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][old][1295585][122299] duration [15.3s], collections [2]/[15.4s], total [15.3s]/[12.2h], memory [13.7gb]->[12.4gb]/[13.8gb], all_pools {[young] [1.1gb]->[2.4mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [12.4gb]->[12.4gb]/[12.5gb]} [2018-01-08T13:32:53,044][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][1295585] overhead, spent [15.3s] collecting in the last [15.4s] [2018-01-08T13:33:16,008][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][old][1295593][122300] duration [14.4s], collections [1]/[15.9s], total [14.4s]/[12.2h], memory [13.6gb]->[12.7gb]/[13.8gb], all_pools {[young] [1gb]->[253.4mb]/[1.1gb]}{[survivor] [131.8mb]->[0b]/[149.7mb]}{[old] [12.4gb]->[12.4gb]/[12.5gb]} [2018-01-08T13:33:16,023][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][1295593] overhead, spent [14.9s] collecting in the last [15.9s] [2018-01-08T13:33:33,060][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][old][1295596][122301] duration [14.7s], collections [1]/[15s], total [14.7s]/[12.2h], memory [13.4gb]->[12.2gb]/[13.8gb], all_pools {[young] [859.2mb]->[2.5mb]/[1.1gb]}{[survivor] [113.3mb]->[0b]/[149.7mb]}{[old] [12.5gb]->[12.1gb]/[12.5gb]} [2018-01-08T13:33:33,060][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][1295596] overhead, spent [14.7s] collecting in the last [15s] [2018-01-08T13:35:32,803][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][old][1295700][122316] duration [15.8s], collections [1]/[16.2s], total [15.8s]/[12.2h], memory [13.4gb]->[12.5gb]/[13.8gb], all_pools {[young] [985.1mb]->[163.6mb]/[1.1gb]}{[survivor] [125mb]->[0b]/[149.7mb]}{[old] [12.4gb]->[12.4gb]/[12.5gb]} [2018-01-08T13:35:32,804][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][1295700] overhead, spent [15.8s] collecting in the last [16.2s] [2018-01-08T13:35:49,828][WARN][o.e.m.j.JvmGcMonitorService] [es-data-m-2] [gc][old][1295703][122317] duration [14.7s], collections [1]/[15s], total [14.7s]/[12.2h], memory [13gb]->[12.1gb]/[13.8gb], all_pools {[young] [448.8mb]->[27mb]/[1.1gb]}{[survivor] [116.6mb]->[0b]/[149.7mb]}{[old] [12.4gb]->[12.1gb]/[12.5gb]} ``` Then we delete some indices, it's 17000, everything is ok. But two days later, the same problem occurs. We have to merge some small indices into a big one. Then we down the number of shards to 9000. It went well for about 3 days. gc problems occurs again. We delete some indices to make it 7000, Now it's ok. How should we improve the gc performace ? We cann't continue to delete the indices. And this is the jvm of a new restart data node. ``` { "_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster_name" : "myesdb", "nodes" : { "pw4f0oWfT9KGSGi84AKm4w" : { "timestamp" : 1515421426980, "name" : "es-data-m-9", "transport_address" : "10.83.56.10:9300", "host" : "10.83.56.10", "ip" : "10.83.56.10:9300", "roles" : [ "data", "ingest" ], "jvm" : { "timestamp" : 1515421426981, "uptime_in_millis" : 11791868, "mem" : { "heap_used_in_bytes" : 10351075064, "heap_used_percent" : 69, "heap_committed_in_bytes" : 14875361280, "heap_max_in_bytes" : 14875361280, "non_heap_used_in_bytes" : 145224888, "non_heap_committed_in_bytes" : 152145920, "pools" : { "young" : { "used_in_bytes" : 988229576, "max_in_bytes" : 1256259584, "peak_used_in_bytes" : 1256259584, "peak_max_in_bytes" : 1256259584 }, "survivor" : { "used_in_bytes" : 154718320, "max_in_bytes" : 157024256, "peak_used_in_bytes" : 157024256, "peak_max_in_bytes" : 157024256 }, "old" : { "used_in_bytes" : 9208127168, "max_in_bytes" : 13462077440, "peak_used_in_bytes" : 13088728336, "peak_max_in_bytes" : 13462077440 } } }, "threads" : { "count" : 272, "peak_count" : 314 }, "gc" : { "collectors" : { "young" : { "collection_count" : 2845, "collection_time_in_millis" : 168440 }, "old" : { "collection_count" : 981, "collection_time_in_millis" : 202767 } } }, "buffer_pools" : { "direct" : { "count" : 274, "used_in_bytes" : 815798639, "total_capacity_in_bytes" : 815798638 }, "mapped" : { "count" : 22171, "used_in_bytes" : 1962403489862, "total_capacity_in_bytes" : 1962403489862 } }, "classes" : { "current_loaded_count" : 12300, "total_loaded_count" : 12511, "total_unloaded_count" : 211 } } } } } ``` ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 8, 2018, 2:28pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/2 "2018-01-08T14:28:32Z")
27K shards sounds like a lot of shards (almost 1.5K per data node). Maybe you have too many shards given the size of your data. 1. How many indices do you have? 2. How much data are you storing in each of your daily indices? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2018, 2:29pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/3 "2018-01-08T14:29:50Z")
Have a look [at this blog post for some general guidelines](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). ---
**Author:** ![gensmusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gensmusic/32/39173_2.png) [@gensmusic](https://discuss.elastic.co/u/gensmusic)\ **Post date:** [January 8, 2018, 2:32pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/4 "2018-01-08T14:32:10Z")
Now we have 7000 shards, about 368 indices. We have 2.5T store size everyday. And we use es 5.5.1 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2018, 2:41pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/5 "2018-01-08T14:41:30Z")
What is your average shard size? How much heap have you got configured for each data node? Are you forcemerging indices no longer being written to (assuming you have time-based indices) in order to reduce the number of segments? ---
**Author:** ![gensmusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gensmusic/32/39173_2.png) [@gensmusic](https://discuss.elastic.co/u/gensmusic)\ **Post date:** [January 8, 2018, 2:46pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/6 "2018-01-08T14:46:07Z")
the size of shard vary from 2G to 400G. It depends on the log data of that service. most of them are less than 100G. I use reindex to merge the small ones into a big one then delete the small ones. I use -Xms14G -Xmx14G to start the elasticsearch. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2018, 2:49pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/7 "2018-01-08T14:49:59Z")
If you are using time-based indices, you might want to look into using the [rollover index API](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/indices-rollover-index.html) to create new indices only when they reach a particular age and/or size. This can save from having to reindex a lot of data and still allow you to handle peaky traffic without certain indices/shards getting too large. This is also described in [this blog post](https://www.elastic.co/blog/managing-time-based-indices-efficiently). ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 8, 2018, 2:50pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/8 "2018-01-08T14:50:45Z")
100G per shard sounds way too big. A size between 20GB-50GB per shard is much more reasonable. So you have a mix between too many shards that are too small (lots of 2GB shards) and also too many that are too big (lots of 100GB+ shards). Indices with lots of data should probably be configured with more primary shards, while indices getting less data should be configured with less primary shards. Also look into the rollover index API like Christian suggested. ---
**Author:** ![gensmusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gensmusic/32/39173_2.png) [@gensmusic](https://discuss.elastic.co/u/gensmusic)\ **Post date:** [January 8, 2018, 2:54pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/9 "2018-01-08T14:54:32Z")
Thanks a lot, it really helps. I will do some research on these articles. And one more question, we use filebeat to write data directly to es. does filebeat support rollover index api ?? ---
**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\ **Post date:** [January 8, 2018, 2:57pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/10 "2018-01-08T14:57:21Z")
The Rollover index API is something you configure into ES directly and whenever an index grows too big (or gets too old), it gets rolled over. Filebeat has no notion whatsoever of that feature, it's transparent for it. ---
**Author:** ![gensmusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gensmusic/32/39173_2.png) [@gensmusic](https://discuss.elastic.co/u/gensmusic)\ **Post date:** [January 8, 2018, 2:58pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/11 "2018-01-08T14:58:38Z")
Oh, I see, I just saw the api. Thanks a lot, I will close the issue for now. Again, thank you very much, guys. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2018, 3:02pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/12 "2018-01-08T15:02:20Z")
If you are not already, you can use [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) to manage the rollover process. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2018, 3:07pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/13 "2018-01-08T15:07:00Z")
It looks like you are using ingest nodes. If so, I would generally also recommend using dedicated nodes for this. ---
**Author:** ![gensmusic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gensmusic/32/39173_2.png) [@gensmusic](https://discuss.elastic.co/u/gensmusic)\ **Post date:** [January 8, 2018, 3:13pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/14 "2018-01-08T15:13:59Z")
Ok, cool, thanks again. I will try curator, and use ingest nodes as dedicated too. Yeah, so happy to known how to do. 🤣 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2018, 3:14pm UTC](https://discuss.elastic.co/t/gc-takes-a-lot-of-time/114539/15 "2018-02-05T15:14:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.