# Elasticsearch garbage collection problem

**URL:** <https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758>\
**Category:** Elasticsearch\
**Created:** [April 27, 2018, 3:54am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758 "2018-04-27T03:54:30Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\
**Post date:** [April 27, 2018, 3:54am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/1 "2018-04-27T03:54:30Z")

</div>

Hi, I have a cluster with four nodes to collect netflow , and every node has 16G RAM. So I set JAVA heap 8G. After working about three or four days the elasticsearch start to garbage collection(about 30 min) ,which cause the ES low performance and drop flow.

this is the log:

```
[2018-04-26T23:34:29,634][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189285] overhead, spent [548ms] collecting in the last [1.2s]
[2018-04-26T23:34:38,988][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189293][127678] duration [1.9s], collections [1]/[2.3s], total [1.9s]/[1.1h], memory [1.8gb]->[1.5gb]/[7.9gb], all_pools {[young] [362.5mb]->[6.5mb]/[532.5mb]}{[survivor] [66.5mb]->[26.1mb]/[66.5mb]}{[old] [1.4gb]->[1.4gb]/[7.3gb]}
[2018-04-26T23:34:38,988][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189293] overhead, spent [1.9s] collecting in the last [2.3s]
[2018-04-26T23:35:06,062][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189320][127701] duration [776ms], collections [1]/[1s], total [776ms]/[1.1h], memory [2.3gb]->[2gb]/[7.9gb], all_pools {[young] [396.6mb]->[10.6mb]/[532.5mb]}{[survivor] [63.4mb]->[40.3mb]/[66.5mb]}{[old] [1.9gb]->[1.9gb]/[7.3gb]}
[2018-04-26T23:35:06,063][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189320] overhead, spent [776ms] collecting in the last [1s]
[2018-04-26T23:35:08,955][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189322][127703] duration [934ms], collections [1]/[1.8s], total [934ms]/[1.1h], memory [2gb]->[2gb]/[7.9gb], all_pools {[young] [8.3mb]->[1.2mb]/[532.5mb]}{[survivor] [66.5mb]->[24.4mb]/[66.5mb]}{[old] [1.9gb]->[2gb]/[7.3gb]}
[2018-04-26T23:35:08,956][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189322] overhead, spent [934ms] collecting in the last [1.8s]
[2018-04-26T23:36:37,090][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189395][127769] duration [3.6s], collections [1]/[4.5s], total [3.6s]/[1.1h], memory [3.2gb]->[3.2gb]/[7.9gb], all_pools {[young] [31mb]->[1.4mb]/[532.5mb]}{[survivor] [61.7mb]->[65.7mb]/[66.5mb]}{[old] [3.1gb]->[3.1gb]/[7.3gb]}
[2018-04-26T23:36:37,090][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189395] overhead, spent [3.6s] collecting in the last [4.5s]
[2018-04-26T23:36:40,152][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189396][127770] duration [2.2s], collections [1]/[3s], total [2.2s]/[1.1h], memory [3.2gb]->[3.2gb]/[7.9gb], all_pools {[young] [1.4mb]->[4.1mb]/[532.5mb]}{[survivor] [65.7mb]->[31mb]/[66.5mb]}{[old] [3.1gb]->[3.2gb]/[7.3gb]}
[2018-04-26T23:36:40,152][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189396] overhead, spent [2.2s] collecting in the last [3s]
[2018-04-26T23:36:48,155][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189400][127774] duration [4s], collections [1]/[4.9s], total [4s]/[1.1h], memory [3.2gb]->[3.3gb]/[7.9gb], all_pools {[young] [220.9kb]->[18.7mb]/[532.5mb]}{[survivor] [65.9mb]->[50.3mb]/[66.5mb]}{[old] [3.2gb]->[3.2gb]/[7.3gb]}
[2018-04-26T23:37:07,968][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189414] overhead, spent [720ms] collecting in the last [1.1s]
[2018-04-26T23:37:09,969][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189416] overhead, spent [351ms] collecting in the last [1s]
[2018-04-26T23:37:13,613][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189419][127790] duration [961ms], collections [1]/[1.6s], total [961ms]/[1.1h], memory [3.8gb]->[3.6gb]/[7.9gb], all_pools {[young] [220.5mb]->[4.8mb]/[532.5mb]}{[survivor] [53.4mb]->[44.2mb]/[66.5mb]}{[old] [3.5gb]->[3.5gb]/[7.3gb]}
[2018-04-26T23:37:13,614][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189419] overhead, spent [961ms] collecting in the last [1.6s]
[2018-04-26T23:37:16,628][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189422][127792] duration [789ms], collections [1]/[1s], total [789ms]/[1.1h], memory [4gb]->[3.6gb]/[7.9gb], all_pools {[young] [450mb]->[3.3mb]/[532.5mb]}{[survivor] [55.3mb]->[64.4mb]/[66.5mb]}{[old] [3.5gb]->[3.6gb]/[7.3gb]}
[2018-04-26T23:37:16,628][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189422] overhead, spent [789ms] collecting in the last [1s]
[2018-04-26T23:37:19,765][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189424][127793] duration [2s], collections [1]/[2.1s], total [2s]/[1.1h], memory [4.1gb]->[3.6gb]/[7.9gb], all_pools {[young] [479.1mb]->[5.8mb]/[532.5mb]}{[survivor] [64.4mb]->[21.9mb]/[66.5mb]}{[old] [3.6gb]->[3.6gb]/[7.3gb]}
[2018-04-26T23:37:19,765][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189424] overhead, spent [2s] collecting in the last [2.1s]
[2018-04-26T23:37:22,767][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189427] overhead, spent [534ms] collecting in the last [1s]
[2018-04-26T23:37:24,889][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189428][127796] duration [1.4s], collections [1]/[2.1s], total [1.4s]/[1.1h], memory [3.9gb]->[3.7gb]/[7.9gb], all_pools {[young] [164.4mb]->[6.6mb]/[532.5mb]}{[survivor] [40.5mb]->[49.5mb]/[66.5mb]}{[old] [3.7gb]->[3.7gb]/[7.3gb]}
[2018-04-26T23:37:24,889][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189428] overhead, spent [1.4s] collecting in the last [2.1s]
[2018-04-26T23:37:43,410][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189443] overhead, spent [992ms] collecting in the last [1s]
[2018-04-26T23:37:48,582][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][young][189447][127809] duration [1.6s], collections [1]/[2.1s], total [1.6s]/[1.1h], memory [4.2gb]->[4gb]/[7.9gb], all_pools {[young] [253.9mb]->[705.4kb]/[532.5mb]}{[survivor] [36.1mb]->[66.5mb]/[66.5mb]}{[old] [3.9gb]->[3.9gb]/[7.3gb]}
[2018-04-26T23:37:48,582][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189447] overhead, spent [1.6s] collecting in the last [2.1s]
[2018-04-26T23:38:11,798][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189470] overhead, spent [603ms] collecting in the last [1s]
[2018-04-26T23:38:16,160][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189474] overhead, spent [377ms] collecting in the last [1.3s]
[2018-04-26T23:38:30,168][INFO][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189488] overhead, spent [260ms] collecting in the last [1s]
[2018-04-26T23:39:14,001][WARN][o.e.m.j.JvmGcMonitorService] [ES4] [gc][189529] overhead, spent [3.2s] collecting in the last [3.5s]

```

the problem puzzle me a lot.  
Is there any solution to improve the problem?  
thank you in advance 🙂

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 27, 2018, 4:06am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/2 "2018-04-27T04:06:26Z")

</div>

Which version?

---

<div class="post-metadata">

**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\
**Post date:** [April 27, 2018, 4:13am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/3 "2018-04-27T04:13:31Z")

</div>

ES:6.2.2

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 27, 2018, 4:30am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/4 "2018-04-27T04:30:11Z")

</div>

How many indices and shards?

---

<div class="post-metadata">

**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\
**Post date:** [April 27, 2018, 4:43am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/5 "2018-04-27T04:43:12Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e299278a5cc60793a48bfdd038dbbbf99edbbfc5.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b0acdabd60e0624d7b387278da743ccc61076950.png)

the every node shard is default: 5

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 27, 2018, 4:59am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/6 "2018-04-27T04:59:02Z")

</div>

You can probably reduce a bit the number of shards. As you have here at most 82gb per index, may be 2 or 3 shards would be enough?

It will reduce the pressure on the nodes IMO.

Another thing you can do in the short term is to add a new node if possible.

---

<div class="post-metadata">

**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\
**Post date:** [April 27, 2018, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/7 "2018-04-27T10:31:13Z")

</div>

Is the REST API correct to set primary 3 shards?

```
PUT _template/logstash
{
  "index_patterns": ["logstash-*"],
  "settings": {
    "number_of_shards": 3
  }
}

```

After running the command, should I restart all the nodes?

thank you very much 🙂

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 27, 2018, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/8 "2018-04-27T10:43:20Z")

</div>

This looks correct but this will only apply to new indices.  
You don't have to restart.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 25, 2018, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-garbage-collection-problem/129758/9 "2018-05-25T10:43:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.