# Elasticsearch heavy garbage collection

**URL:** <https://discuss.elastic.co/t/elasticsearch-heavy-garbage-collection/22398>\
**Category:** Elasticsearch\
**Created:** [February 26, 2015, 3:32am UTC](https://discuss.elastic.co/t/elasticsearch-heavy-garbage-collection/22398 "2015-02-26T03:32:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chris85lang](https://avatars.discourse-cdn.com/v4/letter/c/a587f6/32.png) [@chris85lang](https://discuss.elastic.co/u/chris85lang)\
**Post date:** [February 26, 2015, 3:32am UTC](https://discuss.elastic.co/t/elasticsearch-heavy-garbage-collection/22398/1 "2015-02-26T03:32:44Z")

</div>

Hello,

we use a elasticsearch cluster with two nodes together with logstash and  
kibana. When we receive a large amount of logs (\>200 per sec.)  
elasticsearch becomes unusable slow.

We see a lot of garbage collection going on:  
[2015-02-25 18:27:02,494][WARN][monitor.jvm] [server1] [gc][old][564][30]  
duration [12.4s], collections [1]/[13s], total [12.4s]/[3.5m], memory  
[9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[323.4mb]/[1.1gb]}{[survivor] [65.5mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:27:14,098][INFO][monitor.jvm] [server1] [gc][old][566][31]  
duration [9.2s], collections [1]/[9.8s], total [9.2s]/[3.7m], memory  
[9.7gb]-\>[8.7gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[237.4mb]/[1.1gb]}{[survivor] [65.7mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:27:27,717][WARN][monitor.jvm] [server1] [gc][old][568][32]  
duration [12s], collections [1]/[12.6s], total [12s]/[3.9m], memory  
[9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[275.2mb]/[1.1gb]}{[survivor] [73.1mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:27:39,518][INFO][monitor.jvm] [server1] [gc][old][570][33]  
duration [9.5s], collections [1]/[9.9s], total [9.5s]/[4.1m], memory  
[9.8gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[341mb]/[1.1gb]}{[survivor] [99.7mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:27:49,549][INFO][monitor.jvm] [server1] [gc][old][571][34]  
duration [9s], collections [1]/[10s], total [9s]/[4.2m], memory  
[8.8gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
[341mb]-\>[364.5mb]/[1.1gb]}{[survivor] [0b]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:27:59,113][INFO][monitor.jvm] [server1] [gc][old][572][35]  
duration [8.5s], collections [1]/[9.5s], total [8.5s]/[4.4m], memory  
[8.8gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
[364.5mb]-\>[379.8mb]/[1.1gb]}{[survivor] [0b]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:28:08,713][INFO][monitor.jvm] [server1] [gc][old][574][36]  
duration [8.5s], collections [1]/[8.5s], total [8.5s]/[4.5m], memory  
[9.8gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[420.4mb]/[1.1gb]}{[survivor] [146.1mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:28:18,486][INFO][monitor.jvm] [server1] [gc][old][576][37]  
duration [8.3s], collections [1]/[8.7s], total [8.3s]/[4.6m], memory  
[9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[365.2mb]/[1.1gb]}{[survivor] [66.6mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:28:28,169][INFO][monitor.jvm] [server1] [gc][old][578][38]  
duration [8.3s], collections [1]/[8.6s], total [8.3s]/[4.8m], memory  
[9.7gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[389mb]/[1.1gb]}{[survivor] [88.7mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:28:38,022][INFO][monitor.jvm] [server1] [gc][old][580][39]  
duration [8.6s], collections [1]/[8.8s], total [8.6s]/[4.9m], memory  
[9.7gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[387.2mb]/[1.1gb]}{[survivor] [79mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}  
[2015-02-25 18:28:48,061][INFO][monitor.jvm] [server1] [gc][old][582][40]  
duration [8.2s], collections [1]/[9s], total [8.2s]/[5.1m], memory  
[9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
[1.1gb]-\>[325.4mb]/[1.1gb]}{[survivor] [14.6mb]-\>[0b]/[149.7mb]}{[old]  
[8.5gb]-\>[8.5gb]/[8.5gb]}

In htop we see its mainly one core who is busy all the time:

[http://i.imgur.com/7LDS8h4.png](http://i.imgur.com/7LDS8h4.png)

Any ideas?

Cheers,  
Chris

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b85a0a38-a461-4d14-b188-1a77f5307444%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b85a0a38-a461-4d14-b188-1a77f5307444%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\
**Post date:** [February 26, 2015, 7:59am UTC](https://discuss.elastic.co/t/elasticsearch-heavy-garbage-collection/22398/2 "2015-02-26T07:59:45Z")

</div>

when gc duration more than a second, client application will notice the  
cluster slow down, perhaps find out what are the hot thread?

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

and took a heap dump in elasticsearch node

> <https://stackoverflow.com/questions/407612/how-to-get-a-thread-and-heap-dump-of-a-java-process-on-windows-thats-not-runnin>

and see what's causing it and start investigate from there?

hth

jason

On Thu, Feb 26, 2015 at 11:32 AM, [chris85lang@googlemail.com](mailto:chris85lang@googlemail.com) wrote:

> Hello,
> 
> we use a elasticsearch cluster with two nodes together with logstash and  
> kibana. When we receive a large amount of logs (\>200 per sec.)  
> elasticsearch becomes unusable slow.
> 
> We see a lot of garbage collection going on:  
> [2015-02-25 18:27:02,494][WARN][monitor.jvm] [server1] [gc][old][564][30]  
> duration [12.4s], collections [1]/[13s], total [12.4s]/[3.5m], memory  
> [9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[323.4mb]/[1.1gb]}{[survivor] [65.5mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:27:14,098][INFO][monitor.jvm] [server1] [gc][old][566][31]  
> duration [9.2s], collections [1]/[9.8s], total [9.2s]/[3.7m], memory  
> [9.7gb]-\>[8.7gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[237.4mb]/[1.1gb]}{[survivor] [65.7mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:27:27,717][WARN][monitor.jvm] [server1] [gc][old][568][32]  
> duration [12s], collections [1]/[12.6s], total [12s]/[3.9m], memory  
> [9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[275.2mb]/[1.1gb]}{[survivor] [73.1mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:27:39,518][INFO][monitor.jvm] [server1] [gc][old][570][33]  
> duration [9.5s], collections [1]/[9.9s], total [9.5s]/[4.1m], memory  
> [9.8gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[341mb]/[1.1gb]}{[survivor] [99.7mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:27:49,549][INFO][monitor.jvm] [server1] [gc][old][571][34]  
> duration [9s], collections [1]/[10s], total [9s]/[4.2m], memory  
> [8.8gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
> [341mb]-\>[364.5mb]/[1.1gb]}{[survivor] [0b]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:27:59,113][INFO][monitor.jvm] [server1] [gc][old][572][35]  
> duration [8.5s], collections [1]/[9.5s], total [8.5s]/[4.4m], memory  
> [8.8gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
> [364.5mb]-\>[379.8mb]/[1.1gb]}{[survivor] [0b]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:28:08,713][INFO][monitor.jvm] [server1] [gc][old][574][36]  
> duration [8.5s], collections [1]/[8.5s], total [8.5s]/[4.5m], memory  
> [9.8gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[420.4mb]/[1.1gb]}{[survivor] [146.1mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:28:18,486][INFO][monitor.jvm] [server1] [gc][old][576][37]  
> duration [8.3s], collections [1]/[8.7s], total [8.3s]/[4.6m], memory  
> [9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[365.2mb]/[1.1gb]}{[survivor] [66.6mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:28:28,169][INFO][monitor.jvm] [server1] [gc][old][578][38]  
> duration [8.3s], collections [1]/[8.6s], total [8.3s]/[4.8m], memory  
> [9.7gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[389mb]/[1.1gb]}{[survivor] [88.7mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:28:38,022][INFO][monitor.jvm] [server1] [gc][old][580][39]  
> duration [8.6s], collections [1]/[8.8s], total [8.6s]/[4.9m], memory  
> [9.7gb]-\>[8.9gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[387.2mb]/[1.1gb]}{[survivor] [79mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}  
> [2015-02-25 18:28:48,061][INFO][monitor.jvm] [server1] [gc][old][582][40]  
> duration [8.2s], collections [1]/[9s], total [8.2s]/[5.1m], memory  
> [9.7gb]-\>[8.8gb]/[9.8gb], all\_pools {[young]  
> [1.1gb]-\>[325.4mb]/[1.1gb]}{[survivor] [14.6mb]-\>[0b]/[149.7mb]}{[old]  
> [8.5gb]-\>[8.5gb]/[8.5gb]}
> 
> In htop we see its mainly one core who is busy all the time:
> 
> [http://i.imgur.com/7LDS8h4.png](http://i.imgur.com/7LDS8h4.png)
> 
> Any ideas?
> 
> Cheers,  
> Chris
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/b85a0a38-a461-4d14-b188-1a77f5307444%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b85a0a38-a461-4d14-b188-1a77f5307444%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/b85a0a38-a461-4d14-b188-1a77f5307444%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b85a0a38-a461-4d14-b188-1a77f5307444%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itxFR%3DN0TFFmC0vX-63r\_YSq3-orYu94%2BZ2GTxStVn%2Bm0w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itxFR%3DN0TFFmC0vX-63r_YSq3-orYu94%2BZ2GTxStVn%2Bm0w%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:29am UTC](https://discuss.elastic.co/t/elasticsearch-heavy-garbage-collection/22398/3 "2017-07-06T00:29:56Z")

</div>