# GC issues after upgrading to 0.19

**URL:** <https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405>\
**Category:** Elasticsearch\
**Created:** [July 13, 2012, 1:05pm UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405 "2012-07-13T13:05:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\
**Post date:** [July 13, 2012, 1:05pm UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/1 "2012-07-13T13:05:06Z")

</div>

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.  
The new zone features are really great, but we observed that nodes  
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to  
process. We increased the timeout so nodes don't drop out anymore. But  
still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might cause this?  
We activated mlock (it would be good if there would a message in the  
log saying that it's active) and used the default GC settings, but it  
didn't help.

We didn't have any GC collection issues in the old version. Any ideas  
what might cause this? We did however optimised the indexing process  
so it's much faster. Could this be related to many documents being  
indexed at the same time?

Below the GC log of our servers:  
Thanks,  
Thibaut

r6n1.  
[2012-07-13 12:00:11,642][INFO  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],  
total [787ms]/[7.8m], memory [5gb]-\>[5gb]/[15.9gb], all\_pools {[Code  
Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[49.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
[13.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]-\>[5gb]/[15.8gb]}{[CMS  
Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:00:26,991][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],  
total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[12mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
[14.8mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
[5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:00:29,288][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],  
total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[1.2mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
[2.7mb]-\>[7.7mb]/[16.6mb]}{[CMS Old Gen]  
[5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:00:48,606][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],  
total [7.9s]/[8.1m], memory [5.4gb]-\>[5.2gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[126mb]-\>[364kb]/[133.1mb]}{[Par Survivor Space]  
[7.2mb]-\>[10.7mb]/[16.6mb]}{[CMS Old Gen]  
[5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:01:12,728][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],  
total [21.9s]/[8.4m], memory [5.4gb]-\>[5.4gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[2.5mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
[14.3mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
[5.4gb]-\>[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:01:38,008][INFO  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],  
total [2.5s]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[68.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
[4.8mb]-\>[10mb]/[16.6mb]}{[CMS Old Gen]  
[5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:01:45,613][INFO  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],  
total [702ms]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[100.3mb]-\>[4.8mb]/[133.1mb]}{[Par Survivor Space]  
[8.4mb]-\>[7.2mb]/[16.6mb]}{[CMS Old Gen]  
[5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:02:25,657][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],  
total [23.3s]/[9m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[77.7mb]-\>[4.1mb]/[133.1mb]}{[Par Survivor Space]  
[1.7mb]-\>[6.8mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:04:43,299][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],  
total [2.2m]/[11.2m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[4.1mb]-\>[747.9kb]/[133.1mb]}{[Par Survivor Space]  
[6.8mb]-\>[16.5mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
[2012-07-13 12:04:59,157][WARN  
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],  
total [15.4s]/[11.5m], memory [5.9gb]-\>[6gb]/[15.9gb], all\_pools  
{[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
[747.9kb]-\>[388.7kb]/[133.1mb]}{[Par Survivor Space]  
[16.5mb]-\>[16.6mb]/[16.6mb]}{[CMS Old Gen]  
[5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
r6n2.  
r6n3.  
r6n4.  
r6n5.  
r6n6.  
[2012-07-13 13:10:26,899][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],  
total [20.8s]/[14.1m], memory [8.7gb]-\>[8.6gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[70.7mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
[11.9mb]-\>[8.7mb]/[16.6mb]}{[CMS Old Gen]  
[8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:10:35,548][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],  
total [5.8s]/[14.2m], memory [8.6gb]-\>[8.6gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[18.7kb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
[2.4mb]-\>[5.2mb]/[16.6mb]}{[CMS Old Gen]  
[8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:10:41,899][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],  
total [4.1s]/[14.3m], memory [8.8gb]-\>[8.6gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[110.6mb]-\>[115.6kb]/[133.1mb]}{[Par Survivor Space]  
[6.1mb]-\>[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS  
Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:11:14,189][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],  
total [8.3s]/[14.5m], memory [8.7gb]-\>[8.7gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[439.9kb]-\>[15.2kb]/[133.1mb]}{[Par Survivor Space]  
[6.9mb]-\>[7.1mb]/[16.6mb]}{[CMS Old Gen]  
[8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:11:50,448][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],  
total [30.7s]/[15m], memory [8.8gb]-\>[8.7gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[57.1mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
[2.8mb]-\>[2.4mb]/[16.6mb]}{[CMS Old Gen]  
[8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:12:05,445][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],  
total [10.1s]/[15.2m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[2.4mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
[558.9kb]-\>[300.6kb]/[16.6mb]}{[CMS Old Gen]  
[8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:13:25,340][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],  
total [1.2m]/[16.4m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
{[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[25.5mb]-\>[126.8kb]/[133.1mb]}{[Par Survivor Space]  
[3.9mb]-\>[3.5mb]/[16.6mb]}{[CMS Old Gen]  
[8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
[2012-07-13 13:14:56,007][WARN  
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],  
total [1.2m]/[17.8m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
[78.8mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
[6.3mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
[8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
r6n7.  
r6n8.  
[2012-07-13 12:43:21,850][WARN  
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],  
total [1.1m]/[4.6m], memory [3.9gb]-\>[3.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[33.6mb]-\>[2.9mb]/[133.1mb]}{[Par Survivor Space]  
[11mb]-\>[9.4mb]/[16.6mb]}{[CMS Old Gen]  
[3.8gb]-\>[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
[2012-07-13 12:43:35,199][WARN  
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total  
[3s]/[4.7m], memory [3.9gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
[15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[77.9mb]-\>[1.8mb]/[133.1mb]}{[Par Survivor Space]  
[12.9mb]-\>[9mb]/[16.6mb]}{[CMS Old Gen]  
[3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
[2012-07-13 12:44:02,539][WARN  
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],  
total [6.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[73.3mb]-\>[1mb]/[133.1mb]}{[Par Survivor Space]  
[8.2mb]-\>[8.8mb]/[16.6mb]}{[CMS Old Gen]  
[3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
[2012-07-13 12:44:40,213][WARN  
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total  
[2.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
[15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[124.3mb]-\>[177.4kb]/[133.1mb]}{[Par Survivor Space]  
[9.9mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
[3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
r6n9.  
r6n10.  
r6n11.  
r6n12.  
r6n13.  
r6n14.  
r6n15.  
r6n16.  
r6n17.  
r6n18.  
r6n19.  
r6n20.  
r7n1.  
[2012-07-13 11:01:45,714][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],  
total [4.2s]/[1m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[65mb]-\>[291.8kb]/[133.1mb]}{[Par Survivor Space]  
[1.7mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 11:01:50,902][INFO  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],  
total [4.2s]/[1m], memory [5.8gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[291.8kb]-\>[8.9mb]/[133.1mb]}{[Par Survivor Space]  
[2.2mb]-\>[2.8mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 11:02:00,444][INFO  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],  
total [806ms]/[1.1m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[119mb]-\>[74.9mb]/[133.1mb]}{[Par Survivor Space]  
[915.6kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
[5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 11:02:26,492][INFO  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],  
total [3.7s]/[1.1m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
{[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[776.8kb]-\>[969.6kb]/[133.1mb]}{[Par Survivor Space]  
[449.2kb]-\>[1.5mb]/[16.6mb]}{[CMS Old Gen]  
[5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 11:02:36,529][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],  
total [2.8s]/[1.2m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[125mb]-\>[2.3mb]/[133.1mb]}{[Par Survivor Space]  
[2mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS  
Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 11:02:41,545][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total  
[4.3s]/[1.3m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[2.3mb]-\>[5mb]/[133.1mb]}{[Par Survivor Space]  
[2.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
[5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 11:04:57,122][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],  
total [4.4s]/[1.4m], memory [6.2gb]-\>[6.1gb]/[15.9gb], all\_pools  
{[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
[78.5mb]-\>[616.8kb]/[133.1mb]}{[Par Survivor Space]  
[2.7mb]-\>[925.9kb]/[16.6mb]}{[CMS Old Gen]  
[6.1gb]-\>[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
[2012-07-13 12:21:35,881][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total  
[1.8s]/[3.5m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools {[Code  
Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[129.3mb]-\>[3.4mb]/[133.1mb]}{[Par Survivor Space]  
[10.4mb]-\>[9.7mb]/[16.6mb]}{[CMS Old Gen]  
[6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:21:46,480][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],  
total [4.8s]/[3.6m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[39.7mb]-\>[10.5mb]/[133.1mb]}{[Par Survivor Space]  
[4.1mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
[6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:22:11,706][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],  
total [1.9s]/[3.7m], memory [6.9gb]-\>[6.9gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[27.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
[8.3mb]-\>[6.7mb]/[16.6mb]}{[CMS Old Gen]  
[6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:22:38,482][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],  
total [2.1s]/[3.7m], memory [7gb]-\>[6.9gb]/[15.9gb], all\_pools {[Code  
Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[55.2mb]-\>[295.7kb]/[133.1mb]}{[Par Survivor Space]  
[9.8mb]-\>[15.3mb]/[16.6mb]}{[CMS Old Gen]  
[6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:22:59,114][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],  
total [3.8s]/[3.8m], memory [7.1gb]-\>[6.9gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[131.2mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
[1.6mb]-\>[2.6mb]/[16.6mb]}{[CMS Old Gen]  
[6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:23:11,637][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total  
[1s]/[3.9m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
[15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[63.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
[1mb]-\>[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:23:44,223][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total  
[1.7s]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
[15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[1.1mb]-\>[16.8kb]/[133.1mb]}{[Par Survivor Space]  
[1.4mb]-\>[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:23:49,542][INFO  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],  
total [724ms]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code  
Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[2.5mb]-\>[2.7mb]/[133.1mb]}{[Par Survivor Space]  
[913.9kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:24:48,348][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],  
total [2.2s]/[4.3m], memory [7.1gb]-\>[7.1gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[1.2mb]-\>[1.6mb]/[133.1mb]}{[Par Survivor Space]  
[736.2kb]-\>[783.1kb]/[16.6mb]}{[CMS Old Gen]  
[7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:24:52,591][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total  
[1s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools {[Code Cache]  
[15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[92.1mb]-\>[67.8kb]/[133.1mb]}{[Par Survivor Space]  
[899.7kb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
[7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:24:58,726][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],  
total [3.2s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[120.9mb]-\>[9mb]/[133.1mb]}{[Par Survivor Space]  
[5.1mb]-\>[4.3mb]/[16.6mb]}{[CMS Old Gen]  
[7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:25:19,635][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],  
total [6.8s]/[4.5m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[105.4mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
[1.7mb]-\>[1.9mb]/[16.6mb]}{[CMS Old Gen]  
[7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:25:46,737][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],  
total [5.2s]/[4.6m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[115.8mb]-\>[12.5kb]/[133.1mb]}{[Par Survivor Space]  
[862kb]-\>[548.3kb]/[16.6mb]}{[CMS Old Gen]  
[1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 12:25:53,811][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],  
total [4.7s]/[4.7m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
{[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
[120mb]-\>[2.2mb]/[133.1mb]}{[Par Survivor Space]  
[689.2kb]-\>[680.3kb]/[16.6mb]}{[CMS Old Gen]  
[1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
[2012-07-13 13:07:55,134][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],  
total [5.6s]/[5.6m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[99.4mb]-\>[357.7kb]/[133.1mb]}{[Par Survivor Space]  
[6.7mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:08:06,093][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],  
total [2.3s]/[5.6m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[34.4mb]-\>[572.1kb]/[133.1mb]}{[Par Survivor Space]  
[1.3mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:08:31,880][INFO  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],  
total [3.6s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[26.5mb]-\>[483.4kb]/[133.1mb]}{[Par Survivor Space]  
[3mb]-\>[797.2kb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:08:35,299][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],  
total [3.2s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[483.4kb]-\>[551.4kb]/[133.1mb]}{[Par Survivor Space]  
[797.2kb]-\>[5mb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:08:41,541][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],  
total [2.4s]/[5.8m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[6mb]-\>[28.8kb]/[133.1mb]}{[Par Survivor Space]  
[300.9kb]-\>[374.1kb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:08:48,737][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],  
total [3.3s]/[5.8m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[68.2mb]-\>[50.6kb]/[133.1mb]}{[Par Survivor Space]  
[1.7mb]-\>[1.2mb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:08:55,241][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],  
total [1.1s]/[5.9m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[52.9kb]-\>[16.5kb]/[133.1mb]}{[Par Survivor Space]  
[2.4mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
[5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:09:24,266][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],  
total [1.9s]/[5.9m], memory [5.7gb]-\>[5.7gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[1.1mb]-\>[206.9kb]/[133.1mb]}{[Par Survivor Space]  
[3.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
[5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:09:59,338][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],  
total [9.1s]/[6.1m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[97.6mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
[2.1mb]-\>[1.6mb]/[16.6mb]}{[CMS Old Gen]  
[5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:10:07,579][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],  
total [3.3s]/[6.2m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[58.9mb]-\>[5.8mb]/[133.1mb]}{[Par Survivor Space]  
[1.1mb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS  
Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:10:30,327][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],  
total [6.6s]/[6.3m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[44.2mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
[633.6kb]-\>[685.6kb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:10:40,008][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],  
total [1.9s]/[6.4m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[42.9mb]-\>[174.3kb]/[133.1mb]}{[Par Survivor Space]  
[1.1mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:11:22,592][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],  
total [1.5s]/[6.4m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[50.5mb]-\>[3.3mb]/[133.1mb]}{[Par Survivor Space]  
[2.9mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:11:35,865][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],  
total [10.8s]/[6.6m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[37.6mb]-\>[7.5mb]/[133.1mb]}{[Par Survivor Space]  
[181.5kb]-\>[133.3kb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:11:37,723][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],  
total [1.3s]/[6.6m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[7.5mb]-\>[18.3kb]/[133.1mb]}{[Par Survivor Space]  
[133.3kb]-\>[540kb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:11:47,016][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],  
total [3s]/[6.7m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[63.3mb]-\>[3.6mb]/[133.1mb]}{[Par Survivor Space]  
[1.5mb]-\>[1.3mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:11:55,906][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],  
total [2.3s]/[6.7m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[126.9mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
[1.5mb]-\>[1.4mb]/[16.6mb]}{[CMS Old Gen]  
[5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
[2012-07-13 13:12:23,075][WARN  
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],  
total [1.1s]/[6.8m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
{[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
[1.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
[184.2kb]-\>[237.9kb]/[16.6mb]}{[CMS Old Gen]  
[5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
r7n2.  
r7n3.  
[2012-07-13 10:44:24,127][WARN  
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],  
total [10.2s]/[1m], memory [8.9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
[108.3mb]-\>[24mb]/[133.1mb]}{[Par Survivor Space]  
[10.9mb]-\>[7.3mb]/[16.6mb]}{[CMS Old Gen]  
[8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
[2012-07-13 10:45:02,249][WARN  
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],  
total [36.5s]/[1.6m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
[95mb]-\>[9.3mb]/[133.1mb]}{[Par Survivor Space]  
[11.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
[8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
[2012-07-13 10:45:43,819][WARN  
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],  
total [15.1s]/[1.9m], memory [9.4gb]-\>[9.3gb]/[15.9gb], all\_pools  
{[Code Cache] [12.9mb]-\>[12.9mb]/[48mb]}{[Par Eden Space]  
[121.2mb]-\>[13.1mb]/[133.1mb]}{[Par Survivor Space]  
[7.9mb]-\>[15.5mb]/[16.6mb]}{[CMS Old Gen]  
[9.3gb]-\>[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
[2012-07-13 10:46:01,557][WARN  
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],  
total [10.6s]/[2.1m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools  
{[Code Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
[124.9mb]-\>[309.7kb]/[133.1mb]}{[Par Survivor Space]  
[7.1mb]-\>[7.9mb]/[16.6mb]}{[CMS Old Gen]  
[9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
[2012-07-13 10:49:05,877][WARN  
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total  
[3m]/[5.2m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools {[Code Cache]  
[13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
[95.9mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
[15.7mb]-\>[12mb]/[16.6mb]}{[CMS Old Gen]  
[9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]-\>[50.9mb]/[82mb]}  
r7n4.  
r7n5.  
r7n6.  
r7n7.  
r7n8.  
r7n9.  
r7n10.  
r7n11.  
r7n12.  
r7n13.  
r7n14.  
r7n15.  
r7n16.  
r7n17.  
r7n18.  
[2012-07-13 10:32:32,003][WARN  
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total  
[4.1s]/[9.2s], memory [1.4gb]-\>[1.3gb]/[15.9gb], all\_pools {[Code  
Cache] [10.5mb]-\>[10.5mb]/[48mb]}{[Par Eden Space]  
[59.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
[10.1mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
[1.3gb]-\>[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]-\>[56.3mb]/[82mb]}  
[2012-07-13 10:37:14,787][WARN  
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total  
[2.7m]/[2.9m], memory [1.5gb]-\>[1.5gb]/[15.9gb], all\_pools {[Code  
Cache] [10.6mb]-\>[10.6mb]/[48mb]}{[Par Eden Space]  
[1.3mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
[6.6mb]-\>[13.3mb]/[16.6mb]}{[CMS Old Gen]  
[1.5gb]-\>[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]-\>[56.7mb]/[82mb]}  
r7n19.  
r7n20.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 14, 2012, 11:28am UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/2 "2012-07-14T11:28:46Z")

</div>

You might simply had it before as well, except that in 0.19, this is logged  
now (the logging part is new). Such long GC pauses for the Par New  
generation are strange, please check if you don't end up swapping the JVM  
(and also turn on bootstrap.mlockall, check the installation guide).

Also, where are you running elasticsearch on? Is it AWS? If so, which OS  
are you using?

Last, a bit more details can be obtained by turning on GC logging  
information. You can head to bin/elasticsearch.in.sh, and uncomment  
the -XX:+PrintGCDetails and the path to the GC logging file. What would be  
interesting there is to see the details associated with such long  
collections (and see if its actually user time being spent doing them).

On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz \<  
[thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote:

> Hi,
> 
> we just upgraded our production cluster from 0.17 to 0.18 to 0.19.  
> The new zone features are really great, but we observed that nodes  
> were constantly marked as invalid and drop out.
> 
> As it turns out, the GC sometimes needs more than 2 or 3 minutes to  
> process. We increased the timeout so nodes don't drop out anymore. But  
> still all requests to taht node will hang for a few minutes
> 
> I saw similar posts here in the last 2 weeks. Any idea what might cause  
> this?  
> We activated mlock (it would be good if there would a message in the  
> log saying that it's active) and used the default GC settings, but it  
> didn't help.
> 
> We didn't have any GC collection issues in the old version. Any ideas  
> what might cause this? We did however optimised the indexing process  
> so it's much faster. Could this be related to many documents being  
> indexed at the same time?
> 
> Below the GC log of our servers:  
> Thanks,  
> Thibaut
> 
> r6n1.  
> [2012-07-13 12:00:11,642][INFO  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],  
> total [787ms]/[7.8m], memory [5gb]-\>[5gb]/[15.9gb], all\_pools {[Code  
> Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [49.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> [13.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]-\>[5gb]/[15.8gb]}{[CMS  
> Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:00:26,991][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],  
> total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [12mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> [14.8mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:00:29,288][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],  
> total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [1.2mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> [2.7mb]-\>[7.7mb]/[16.6mb]}{[CMS Old Gen]  
> [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:00:48,606][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],  
> total [7.9s]/[8.1m], memory [5.4gb]-\>[5.2gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [126mb]-\>[364kb]/[133.1mb]}{[Par Survivor Space]  
> [7.2mb]-\>[10.7mb]/[16.6mb]}{[CMS Old Gen]  
> [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:01:12,728][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],  
> total [21.9s]/[8.4m], memory [5.4gb]-\>[5.4gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [2.5mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> [14.3mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> [5.4gb]-\>[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:01:38,008][INFO  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],  
> total [2.5s]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [68.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> [4.8mb]-\>[10mb]/[16.6mb]}{[CMS Old Gen]  
> [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:01:45,613][INFO  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],  
> total [702ms]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [100.3mb]-\>[4.8mb]/[133.1mb]}{[Par Survivor Space]  
> [8.4mb]-\>[7.2mb]/[16.6mb]}{[CMS Old Gen]  
> [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:02:25,657][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],  
> total [23.3s]/[9m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [77.7mb]-\>[4.1mb]/[133.1mb]}{[Par Survivor Space]  
> [1.7mb]-\>[6.8mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:04:43,299][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],  
> total [2.2m]/[11.2m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [4.1mb]-\>[747.9kb]/[133.1mb]}{[Par Survivor Space]  
> [6.8mb]-\>[16.5mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> [2012-07-13 12:04:59,157][WARN  
> ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> [gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],  
> total [15.4s]/[11.5m], memory [5.9gb]-\>[6gb]/[15.9gb], all\_pools  
> {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> [747.9kb]-\>[388.7kb]/[133.1mb]}{[Par Survivor Space]  
> [16.5mb]-\>[16.6mb]/[16.6mb]}{[CMS Old Gen]  
> [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> r6n2.  
> r6n3.  
> r6n4.  
> r6n5.  
> r6n6.  
> [2012-07-13 13:10:26,899][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],  
> total [20.8s]/[14.1m], memory [8.7gb]-\>[8.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [70.7mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> [11.9mb]-\>[8.7mb]/[16.6mb]}{[CMS Old Gen]  
> [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:10:35,548][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],  
> total [5.8s]/[14.2m], memory [8.6gb]-\>[8.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [18.7kb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> [2.4mb]-\>[5.2mb]/[16.6mb]}{[CMS Old Gen]  
> [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:10:41,899][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],  
> total [4.1s]/[14.3m], memory [8.8gb]-\>[8.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [110.6mb]-\>[115.6kb]/[133.1mb]}{[Par Survivor Space]  
> [6.1mb]-\>[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS  
> Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:11:14,189][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],  
> total [8.3s]/[14.5m], memory [8.7gb]-\>[8.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [439.9kb]-\>[15.2kb]/[133.1mb]}{[Par Survivor Space]  
> [6.9mb]-\>[7.1mb]/[16.6mb]}{[CMS Old Gen]  
> [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:11:50,448][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],  
> total [30.7s]/[15m], memory [8.8gb]-\>[8.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [57.1mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> [2.8mb]-\>[2.4mb]/[16.6mb]}{[CMS Old Gen]  
> [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:12:05,445][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],  
> total [10.1s]/[15.2m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [2.4mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> [558.9kb]-\>[300.6kb]/[16.6mb]}{[CMS Old Gen]  
> [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:13:25,340][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],  
> total [1.2m]/[16.4m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [25.5mb]-\>[126.8kb]/[133.1mb]}{[Par Survivor Space]  
> [3.9mb]-\>[3.5mb]/[16.6mb]}{[CMS Old Gen]  
> [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> [2012-07-13 13:14:56,007][WARN  
> ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> [gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],  
> total [1.2m]/[17.8m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> [78.8mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> [6.3mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> r6n7.  
> r6n8.  
> [2012-07-13 12:43:21,850][WARN  
> ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> [gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],  
> total [1.1m]/[4.6m], memory [3.9gb]-\>[3.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [33.6mb]-\>[2.9mb]/[133.1mb]}{[Par Survivor Space]  
> [11mb]-\>[9.4mb]/[16.6mb]}{[CMS Old Gen]  
> [3.8gb]-\>[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> [2012-07-13 12:43:35,199][WARN  
> ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> [gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total  
> [3s]/[4.7m], memory [3.9gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [77.9mb]-\>[1.8mb]/[133.1mb]}{[Par Survivor Space]  
> [12.9mb]-\>[9mb]/[16.6mb]}{[CMS Old Gen]  
> [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> [2012-07-13 12:44:02,539][WARN  
> ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> [gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],  
> total [6.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [73.3mb]-\>[1mb]/[133.1mb]}{[Par Survivor Space]  
> [8.2mb]-\>[8.8mb]/[16.6mb]}{[CMS Old Gen]  
> [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> [2012-07-13 12:44:40,213][WARN  
> ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> [gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total  
> [2.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [124.3mb]-\>[177.4kb]/[133.1mb]}{[Par Survivor Space]  
> [9.9mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> r6n9.  
> r6n10.  
> r6n11.  
> r6n12.  
> r6n13.  
> r6n14.  
> r6n15.  
> r6n16.  
> r6n17.  
> r6n18.  
> r6n19.  
> r6n20.  
> r7n1.  
> [2012-07-13 11:01:45,714][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],  
> total [4.2s]/[1m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [65mb]-\>[291.8kb]/[133.1mb]}{[Par Survivor Space]  
> [1.7mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 11:01:50,902][INFO  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],  
> total [4.2s]/[1m], memory [5.8gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [291.8kb]-\>[8.9mb]/[133.1mb]}{[Par Survivor Space]  
> [2.2mb]-\>[2.8mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 11:02:00,444][INFO  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],  
> total [806ms]/[1.1m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [119mb]-\>[74.9mb]/[133.1mb]}{[Par Survivor Space]  
> [915.6kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
> [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 11:02:26,492][INFO  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],  
> total [3.7s]/[1.1m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [776.8kb]-\>[969.6kb]/[133.1mb]}{[Par Survivor Space]  
> [449.2kb]-\>[1.5mb]/[16.6mb]}{[CMS Old Gen]  
> [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 11:02:36,529][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],  
> total [2.8s]/[1.2m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [125mb]-\>[2.3mb]/[133.1mb]}{[Par Survivor Space]  
> [2mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS  
> Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 11:02:41,545][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total  
> [4.3s]/[1.3m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [2.3mb]-\>[5mb]/[133.1mb]}{[Par Survivor Space]  
> [2.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 11:04:57,122][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],  
> total [4.4s]/[1.4m], memory [6.2gb]-\>[6.1gb]/[15.9gb], all\_pools  
> {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> [78.5mb]-\>[616.8kb]/[133.1mb]}{[Par Survivor Space]  
> [2.7mb]-\>[925.9kb]/[16.6mb]}{[CMS Old Gen]  
> [6.1gb]-\>[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> [2012-07-13 12:21:35,881][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total  
> [1.8s]/[3.5m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools {[Code  
> Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [129.3mb]-\>[3.4mb]/[133.1mb]}{[Par Survivor Space]  
> [10.4mb]-\>[9.7mb]/[16.6mb]}{[CMS Old Gen]  
> [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:21:46,480][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],  
> total [4.8s]/[3.6m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [39.7mb]-\>[10.5mb]/[133.1mb]}{[Par Survivor Space]  
> [4.1mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:22:11,706][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],  
> total [1.9s]/[3.7m], memory [6.9gb]-\>[6.9gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [27.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> [8.3mb]-\>[6.7mb]/[16.6mb]}{[CMS Old Gen]  
> [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:22:38,482][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],  
> total [2.1s]/[3.7m], memory [7gb]-\>[6.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [55.2mb]-\>[295.7kb]/[133.1mb]}{[Par Survivor Space]  
> [9.8mb]-\>[15.3mb]/[16.6mb]}{[CMS Old Gen]  
> [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:22:59,114][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],  
> total [3.8s]/[3.8m], memory [7.1gb]-\>[6.9gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [131.2mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> [1.6mb]-\>[2.6mb]/[16.6mb]}{[CMS Old Gen]  
> [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:23:11,637][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total  
> [1s]/[3.9m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [63.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> [1mb]-\>[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:23:44,223][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total  
> [1.7s]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [1.1mb]-\>[16.8kb]/[133.1mb]}{[Par Survivor Space]  
> [1.4mb]-\>[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:23:49,542][INFO  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],  
> total [724ms]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code  
> Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [2.5mb]-\>[2.7mb]/[133.1mb]}{[Par Survivor Space]  
> [913.9kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:24:48,348][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],  
> total [2.2s]/[4.3m], memory [7.1gb]-\>[7.1gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [1.2mb]-\>[1.6mb]/[133.1mb]}{[Par Survivor Space]  
> [736.2kb]-\>[783.1kb]/[16.6mb]}{[CMS Old Gen]  
> [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:24:52,591][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total  
> [1s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools {[Code Cache]  
> [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [92.1mb]-\>[67.8kb]/[133.1mb]}{[Par Survivor Space]  
> [899.7kb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
> [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:24:58,726][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],  
> total [3.2s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [120.9mb]-\>[9mb]/[133.1mb]}{[Par Survivor Space]  
> [5.1mb]-\>[4.3mb]/[16.6mb]}{[CMS Old Gen]  
> [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:25:19,635][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],  
> total [6.8s]/[4.5m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [105.4mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> [1.7mb]-\>[1.9mb]/[16.6mb]}{[CMS Old Gen]  
> [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:25:46,737][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],  
> total [5.2s]/[4.6m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [115.8mb]-\>[12.5kb]/[133.1mb]}{[Par Survivor Space]  
> [862kb]-\>[548.3kb]/[16.6mb]}{[CMS Old Gen]  
> [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 12:25:53,811][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],  
> total [4.7s]/[4.7m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> [120mb]-\>[2.2mb]/[133.1mb]}{[Par Survivor Space]  
> [689.2kb]-\>[680.3kb]/[16.6mb]}{[CMS Old Gen]  
> [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> [2012-07-13 13:07:55,134][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],  
> total [5.6s]/[5.6m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [99.4mb]-\>[357.7kb]/[133.1mb]}{[Par Survivor Space]  
> [6.7mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:08:06,093][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],  
> total [2.3s]/[5.6m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [34.4mb]-\>[572.1kb]/[133.1mb]}{[Par Survivor Space]  
> [1.3mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:08:31,880][INFO  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],  
> total [3.6s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [26.5mb]-\>[483.4kb]/[133.1mb]}{[Par Survivor Space]  
> [3mb]-\>[797.2kb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:08:35,299][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],  
> total [3.2s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [483.4kb]-\>[551.4kb]/[133.1mb]}{[Par Survivor Space]  
> [797.2kb]-\>[5mb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:08:41,541][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],  
> total [2.4s]/[5.8m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [6mb]-\>[28.8kb]/[133.1mb]}{[Par Survivor Space]  
> [300.9kb]-\>[374.1kb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:08:48,737][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],  
> total [3.3s]/[5.8m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [68.2mb]-\>[50.6kb]/[133.1mb]}{[Par Survivor Space]  
> [1.7mb]-\>[1.2mb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:08:55,241][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],  
> total [1.1s]/[5.9m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [52.9kb]-\>[16.5kb]/[133.1mb]}{[Par Survivor Space]  
> [2.4mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:09:24,266][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],  
> total [1.9s]/[5.9m], memory [5.7gb]-\>[5.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [1.1mb]-\>[206.9kb]/[133.1mb]}{[Par Survivor Space]  
> [3.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:09:59,338][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],  
> total [9.1s]/[6.1m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [97.6mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> [2.1mb]-\>[1.6mb]/[16.6mb]}{[CMS Old Gen]  
> [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:10:07,579][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],  
> total [3.3s]/[6.2m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [58.9mb]-\>[5.8mb]/[133.1mb]}{[Par Survivor Space]  
> [1.1mb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS  
> Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:10:30,327][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],  
> total [6.6s]/[6.3m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [44.2mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> [633.6kb]-\>[685.6kb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:10:40,008][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],  
> total [1.9s]/[6.4m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [42.9mb]-\>[174.3kb]/[133.1mb]}{[Par Survivor Space]  
> [1.1mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:11:22,592][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],  
> total [1.5s]/[6.4m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [50.5mb]-\>[3.3mb]/[133.1mb]}{[Par Survivor Space]  
> [2.9mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:11:35,865][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],  
> total [10.8s]/[6.6m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [37.6mb]-\>[7.5mb]/[133.1mb]}{[Par Survivor Space]  
> [181.5kb]-\>[133.3kb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:11:37,723][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],  
> total [1.3s]/[6.6m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [7.5mb]-\>[18.3kb]/[133.1mb]}{[Par Survivor Space]  
> [133.3kb]-\>[540kb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:11:47,016][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],  
> total [3s]/[6.7m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [63.3mb]-\>[3.6mb]/[133.1mb]}{[Par Survivor Space]  
> [1.5mb]-\>[1.3mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:11:55,906][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],  
> total [2.3s]/[6.7m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [126.9mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> [1.5mb]-\>[1.4mb]/[16.6mb]}{[CMS Old Gen]  
> [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> [2012-07-13 13:12:23,075][WARN  
> ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> [gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],  
> total [1.1s]/[6.8m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> [1.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> [184.2kb]-\>[237.9kb]/[16.6mb]}{[CMS Old Gen]  
> [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> r7n2.  
> r7n3.  
> [2012-07-13 10:44:24,127][WARN  
> ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> [gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],  
> total [10.2s]/[1m], memory [8.9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> [108.3mb]-\>[24mb]/[133.1mb]}{[Par Survivor Space]  
> [10.9mb]-\>[7.3mb]/[16.6mb]}{[CMS Old Gen]  
> [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> [2012-07-13 10:45:02,249][WARN  
> ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> [gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],  
> total [36.5s]/[1.6m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> [95mb]-\>[9.3mb]/[133.1mb]}{[Par Survivor Space]  
> [11.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> [2012-07-13 10:45:43,819][WARN  
> ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> [gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],  
> total [15.1s]/[1.9m], memory [9.4gb]-\>[9.3gb]/[15.9gb], all\_pools  
> {[Code Cache] [12.9mb]-\>[12.9mb]/[48mb]}{[Par Eden Space]  
> [121.2mb]-\>[13.1mb]/[133.1mb]}{[Par Survivor Space]  
> [7.9mb]-\>[15.5mb]/[16.6mb]}{[CMS Old Gen]  
> [9.3gb]-\>[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> [2012-07-13 10:46:01,557][WARN  
> ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> [gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],  
> total [10.6s]/[2.1m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools  
> {[Code Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> [124.9mb]-\>[309.7kb]/[133.1mb]}{[Par Survivor Space]  
> [7.1mb]-\>[7.9mb]/[16.6mb]}{[CMS Old Gen]  
> [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> [2012-07-13 10:49:05,877][WARN  
> ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> [gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total  
> [3m]/[5.2m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools {[Code Cache]  
> [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> [95.9mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> [15.7mb]-\>[12mb]/[16.6mb]}{[CMS Old Gen]  
> [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]-\>[50.9mb]/[82mb]}  
> r7n4.  
> r7n5.  
> r7n6.  
> r7n7.  
> r7n8.  
> r7n9.  
> r7n10.  
> r7n11.  
> r7n12.  
> r7n13.  
> r7n14.  
> r7n15.  
> r7n16.  
> r7n17.  
> r7n18.  
> [2012-07-13 10:32:32,003][WARN  
> ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> [gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total  
> [4.1s]/[9.2s], memory [1.4gb]-\>[1.3gb]/[15.9gb], all\_pools {[Code  
> Cache] [10.5mb]-\>[10.5mb]/[48mb]}{[Par Eden Space]  
> [59.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> [10.1mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> [1.3gb]-\>[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]-\>[56.3mb]/[82mb]}  
> [2012-07-13 10:37:14,787][WARN  
> ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> [gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total  
> [2.7m]/[2.9m], memory [1.5gb]-\>[1.5gb]/[15.9gb], all\_pools {[Code  
> Cache] [10.6mb]-\>[10.6mb]/[48mb]}{[Par Eden Space]  
> [1.3mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> [6.6mb]-\>[13.3mb]/[16.6mb]}{[CMS Old Gen]  
> [1.5gb]-\>[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]-\>[56.7mb]/[82mb]}  
> r7n19.  
> r7n20.

---

<div class="post-metadata">

**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\
**Post date:** [July 16, 2012, 11:07am UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/3 "2012-07-16T11:07:49Z")

</div>

Hi Shay,

Swap is turned off and no swap partition exists, bootstrap.mlockall is  
enabled. They are all dedicated servers running on ubuntu 11.04, java  
"1.6.0\_24"  
We will activate the extended GC logging and I will post the results.

The problem seems to mostly occur when we restart the cluster. So I  
suspect it to to be related to a relatively high indexation volume  
when the cluster is newly started.

Thanks,  
Thibaut

On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> You might simply had it before as well, except that in 0.19, this is logged  
> now (the logging part is new). Such long GC pauses for the Par New  
> generation are strange, please check if you don't end up swapping the JVM  
> (and also turn on bootstrap.mlockall, check the installation guide).
> 
> Also, where are you running elasticsearch on? Is it AWS? If so, which OS are  
> you using?
> 
> Last, a bit more details can be obtained by turning on GC logging  
> information. You can head to bin/elasticsearch.in.sh, and uncomment the  
> -XX:+PrintGCDetails and the path to the GC logging file. What would be  
> interesting there is to see the details associated with such long  
> collections (and see if its actually user time being spent doing them).
> 
> On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz  
> [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote:
> 
> > Hi,
> > 
> > we just upgraded our production cluster from 0.17 to 0.18 to 0.19.  
> > The new zone features are really great, but we observed that nodes  
> > were constantly marked as invalid and drop out.
> > 
> > As it turns out, the GC sometimes needs more than 2 or 3 minutes to  
> > process. We increased the timeout so nodes don't drop out anymore. But  
> > still all requests to taht node will hang for a few minutes
> > 
> > I saw similar posts here in the last 2 weeks. Any idea what might cause  
> > this?  
> > We activated mlock (it would be good if there would a message in the  
> > log saying that it's active) and used the default GC settings, but it  
> > didn't help.
> > 
> > We didn't have any GC collection issues in the old version. Any ideas  
> > what might cause this? We did however optimised the indexing process  
> > so it's much faster. Could this be related to many documents being  
> > indexed at the same time?
> > 
> > Below the GC log of our servers:  
> > Thanks,  
> > Thibaut
> > 
> > r6n1.  
> > [2012-07-13 12:00:11,642][INFO  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],  
> > total [787ms]/[7.8m], memory [5gb]-\>[5gb]/[15.9gb], all\_pools {[Code  
> > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [49.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [13.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]-\>[5gb]/[15.8gb]}{[CMS  
> > Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:00:26,991][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],  
> > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [12mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [14.8mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:00:29,288][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],  
> > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [1.2mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.7mb]-\>[7.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:00:48,606][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],  
> > total [7.9s]/[8.1m], memory [5.4gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [126mb]-\>[364kb]/[133.1mb]}{[Par Survivor Space]  
> > [7.2mb]-\>[10.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:01:12,728][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],  
> > total [21.9s]/[8.4m], memory [5.4gb]-\>[5.4gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [2.5mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [14.3mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.4gb]-\>[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:01:38,008][INFO  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],  
> > total [2.5s]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [68.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [4.8mb]-\>[10mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:01:45,613][INFO  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],  
> > total [702ms]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [100.3mb]-\>[4.8mb]/[133.1mb]}{[Par Survivor Space]  
> > [8.4mb]-\>[7.2mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:02:25,657][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],  
> > total [23.3s]/[9m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [77.7mb]-\>[4.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [1.7mb]-\>[6.8mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:04:43,299][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],  
> > total [2.2m]/[11.2m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [4.1mb]-\>[747.9kb]/[133.1mb]}{[Par Survivor Space]  
> > [6.8mb]-\>[16.5mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > [2012-07-13 12:04:59,157][WARN  
> > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > [gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],  
> > total [15.4s]/[11.5m], memory [5.9gb]-\>[6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > [747.9kb]-\>[388.7kb]/[133.1mb]}{[Par Survivor Space]  
> > [16.5mb]-\>[16.6mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > r6n2.  
> > r6n3.  
> > r6n4.  
> > r6n5.  
> > r6n6.  
> > [2012-07-13 13:10:26,899][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],  
> > total [20.8s]/[14.1m], memory [8.7gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [70.7mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [11.9mb]-\>[8.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:10:35,548][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],  
> > total [5.8s]/[14.2m], memory [8.6gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [18.7kb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.4mb]-\>[5.2mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:10:41,899][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],  
> > total [4.1s]/[14.3m], memory [8.8gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [110.6mb]-\>[115.6kb]/[133.1mb]}{[Par Survivor Space]  
> > [6.1mb]-\>[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS  
> > Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:11:14,189][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],  
> > total [8.3s]/[14.5m], memory [8.7gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [439.9kb]-\>[15.2kb]/[133.1mb]}{[Par Survivor Space]  
> > [6.9mb]-\>[7.1mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:11:50,448][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],  
> > total [30.7s]/[15m], memory [8.8gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [57.1mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.8mb]-\>[2.4mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:12:05,445][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],  
> > total [10.1s]/[15.2m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [2.4mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [558.9kb]-\>[300.6kb]/[16.6mb]}{[CMS Old Gen]  
> > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:13:25,340][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],  
> > total [1.2m]/[16.4m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [25.5mb]-\>[126.8kb]/[133.1mb]}{[Par Survivor Space]  
> > [3.9mb]-\>[3.5mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > [2012-07-13 13:14:56,007][WARN  
> > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > [gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],  
> > total [1.2m]/[17.8m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > [78.8mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > [6.3mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > r6n7.  
> > r6n8.  
> > [2012-07-13 12:43:21,850][WARN  
> > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > [gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],  
> > total [1.1m]/[4.6m], memory [3.9gb]-\>[3.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [33.6mb]-\>[2.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [11mb]-\>[9.4mb]/[16.6mb]}{[CMS Old Gen]  
> > [3.8gb]-\>[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > [2012-07-13 12:43:35,199][WARN  
> > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > [gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total  
> > [3s]/[4.7m], memory [3.9gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [77.9mb]-\>[1.8mb]/[133.1mb]}{[Par Survivor Space]  
> > [12.9mb]-\>[9mb]/[16.6mb]}{[CMS Old Gen]  
> > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > [2012-07-13 12:44:02,539][WARN  
> > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > [gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],  
> > total [6.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [73.3mb]-\>[1mb]/[133.1mb]}{[Par Survivor Space]  
> > [8.2mb]-\>[8.8mb]/[16.6mb]}{[CMS Old Gen]  
> > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > [2012-07-13 12:44:40,213][WARN  
> > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > [gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total  
> > [2.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [124.3mb]-\>[177.4kb]/[133.1mb]}{[Par Survivor Space]  
> > [9.9mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > r6n9.  
> > r6n10.  
> > r6n11.  
> > r6n12.  
> > r6n13.  
> > r6n14.  
> > r6n15.  
> > r6n16.  
> > r6n17.  
> > r6n18.  
> > r6n19.  
> > r6n20.  
> > r7n1.  
> > [2012-07-13 11:01:45,714][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],  
> > total [4.2s]/[1m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [65mb]-\>[291.8kb]/[133.1mb]}{[Par Survivor Space]  
> > [1.7mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 11:01:50,902][INFO  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],  
> > total [4.2s]/[1m], memory [5.8gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [291.8kb]-\>[8.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.2mb]-\>[2.8mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 11:02:00,444][INFO  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],  
> > total [806ms]/[1.1m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [119mb]-\>[74.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [915.6kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 11:02:26,492][INFO  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],  
> > total [3.7s]/[1.1m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [776.8kb]-\>[969.6kb]/[133.1mb]}{[Par Survivor Space]  
> > [449.2kb]-\>[1.5mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 11:02:36,529][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],  
> > total [2.8s]/[1.2m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [125mb]-\>[2.3mb]/[133.1mb]}{[Par Survivor Space]  
> > [2mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS  
> > Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 11:02:41,545][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total  
> > [4.3s]/[1.3m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [2.3mb]-\>[5mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 11:04:57,122][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],  
> > total [4.4s]/[1.4m], memory [6.2gb]-\>[6.1gb]/[15.9gb], all\_pools  
> > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > [78.5mb]-\>[616.8kb]/[133.1mb]}{[Par Survivor Space]  
> > [2.7mb]-\>[925.9kb]/[16.6mb]}{[CMS Old Gen]  
> > [6.1gb]-\>[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > [2012-07-13 12:21:35,881][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total  
> > [1.8s]/[3.5m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools {[Code  
> > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [129.3mb]-\>[3.4mb]/[133.1mb]}{[Par Survivor Space]  
> > [10.4mb]-\>[9.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:21:46,480][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],  
> > total [4.8s]/[3.6m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [39.7mb]-\>[10.5mb]/[133.1mb]}{[Par Survivor Space]  
> > [4.1mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:22:11,706][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],  
> > total [1.9s]/[3.7m], memory [6.9gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [27.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [8.3mb]-\>[6.7mb]/[16.6mb]}{[CMS Old Gen]  
> > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:22:38,482][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],  
> > total [2.1s]/[3.7m], memory [7gb]-\>[6.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [55.2mb]-\>[295.7kb]/[133.1mb]}{[Par Survivor Space]  
> > [9.8mb]-\>[15.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:22:59,114][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],  
> > total [3.8s]/[3.8m], memory [7.1gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [131.2mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [1.6mb]-\>[2.6mb]/[16.6mb]}{[CMS Old Gen]  
> > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:23:11,637][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total  
> > [1s]/[3.9m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [63.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [1mb]-\>[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:23:44,223][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total  
> > [1.7s]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [1.1mb]-\>[16.8kb]/[133.1mb]}{[Par Survivor Space]  
> > [1.4mb]-\>[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:23:49,542][INFO  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],  
> > total [724ms]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code  
> > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [2.5mb]-\>[2.7mb]/[133.1mb]}{[Par Survivor Space]  
> > [913.9kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:24:48,348][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],  
> > total [2.2s]/[4.3m], memory [7.1gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [1.2mb]-\>[1.6mb]/[133.1mb]}{[Par Survivor Space]  
> > [736.2kb]-\>[783.1kb]/[16.6mb]}{[CMS Old Gen]  
> > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:24:52,591][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total  
> > [1s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools {[Code Cache]  
> > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [92.1mb]-\>[67.8kb]/[133.1mb]}{[Par Survivor Space]  
> > [899.7kb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
> > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:24:58,726][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],  
> > total [3.2s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [120.9mb]-\>[9mb]/[133.1mb]}{[Par Survivor Space]  
> > [5.1mb]-\>[4.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:25:19,635][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],  
> > total [6.8s]/[4.5m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [105.4mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > [1.7mb]-\>[1.9mb]/[16.6mb]}{[CMS Old Gen]  
> > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:25:46,737][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],  
> > total [5.2s]/[4.6m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [115.8mb]-\>[12.5kb]/[133.1mb]}{[Par Survivor Space]  
> > [862kb]-\>[548.3kb]/[16.6mb]}{[CMS Old Gen]  
> > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 12:25:53,811][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],  
> > total [4.7s]/[4.7m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > [120mb]-\>[2.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [689.2kb]-\>[680.3kb]/[16.6mb]}{[CMS Old Gen]  
> > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > [2012-07-13 13:07:55,134][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],  
> > total [5.6s]/[5.6m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [99.4mb]-\>[357.7kb]/[133.1mb]}{[Par Survivor Space]  
> > [6.7mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:08:06,093][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],  
> > total [2.3s]/[5.6m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [34.4mb]-\>[572.1kb]/[133.1mb]}{[Par Survivor Space]  
> > [1.3mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:08:31,880][INFO  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],  
> > total [3.6s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [26.5mb]-\>[483.4kb]/[133.1mb]}{[Par Survivor Space]  
> > [3mb]-\>[797.2kb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:08:35,299][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],  
> > total [3.2s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [483.4kb]-\>[551.4kb]/[133.1mb]}{[Par Survivor Space]  
> > [797.2kb]-\>[5mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:08:41,541][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],  
> > total [2.4s]/[5.8m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [6mb]-\>[28.8kb]/[133.1mb]}{[Par Survivor Space]  
> > [300.9kb]-\>[374.1kb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:08:48,737][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],  
> > total [3.3s]/[5.8m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [68.2mb]-\>[50.6kb]/[133.1mb]}{[Par Survivor Space]  
> > [1.7mb]-\>[1.2mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:08:55,241][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],  
> > total [1.1s]/[5.9m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [52.9kb]-\>[16.5kb]/[133.1mb]}{[Par Survivor Space]  
> > [2.4mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:09:24,266][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],  
> > total [1.9s]/[5.9m], memory [5.7gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [1.1mb]-\>[206.9kb]/[133.1mb]}{[Par Survivor Space]  
> > [3.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:09:59,338][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],  
> > total [9.1s]/[6.1m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [97.6mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.1mb]-\>[1.6mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:10:07,579][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],  
> > total [3.3s]/[6.2m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [58.9mb]-\>[5.8mb]/[133.1mb]}{[Par Survivor Space]  
> > [1.1mb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS  
> > Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:10:30,327][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],  
> > total [6.6s]/[6.3m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [44.2mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [633.6kb]-\>[685.6kb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:10:40,008][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],  
> > total [1.9s]/[6.4m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [42.9mb]-\>[174.3kb]/[133.1mb]}{[Par Survivor Space]  
> > [1.1mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:11:22,592][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],  
> > total [1.5s]/[6.4m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [50.5mb]-\>[3.3mb]/[133.1mb]}{[Par Survivor Space]  
> > [2.9mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:11:35,865][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],  
> > total [10.8s]/[6.6m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [37.6mb]-\>[7.5mb]/[133.1mb]}{[Par Survivor Space]  
> > [181.5kb]-\>[133.3kb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:11:37,723][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],  
> > total [1.3s]/[6.6m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [7.5mb]-\>[18.3kb]/[133.1mb]}{[Par Survivor Space]  
> > [133.3kb]-\>[540kb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:11:47,016][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],  
> > total [3s]/[6.7m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [63.3mb]-\>[3.6mb]/[133.1mb]}{[Par Survivor Space]  
> > [1.5mb]-\>[1.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:11:55,906][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],  
> > total [2.3s]/[6.7m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [126.9mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [1.5mb]-\>[1.4mb]/[16.6mb]}{[CMS Old Gen]  
> > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > [2012-07-13 13:12:23,075][WARN  
> > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > [gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],  
> > total [1.1s]/[6.8m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > [1.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > [184.2kb]-\>[237.9kb]/[16.6mb]}{[CMS Old Gen]  
> > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > r7n2.  
> > r7n3.  
> > [2012-07-13 10:44:24,127][WARN  
> > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > [gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],  
> > total [10.2s]/[1m], memory [8.9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > [108.3mb]-\>[24mb]/[133.1mb]}{[Par Survivor Space]  
> > [10.9mb]-\>[7.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > [2012-07-13 10:45:02,249][WARN  
> > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > [gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],  
> > total [36.5s]/[1.6m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > [95mb]-\>[9.3mb]/[133.1mb]}{[Par Survivor Space]  
> > [11.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > [2012-07-13 10:45:43,819][WARN  
> > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > [gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],  
> > total [15.1s]/[1.9m], memory [9.4gb]-\>[9.3gb]/[15.9gb], all\_pools  
> > {[Code Cache] [12.9mb]-\>[12.9mb]/[48mb]}{[Par Eden Space]  
> > [121.2mb]-\>[13.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [7.9mb]-\>[15.5mb]/[16.6mb]}{[CMS Old Gen]  
> > [9.3gb]-\>[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > [2012-07-13 10:46:01,557][WARN  
> > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > [gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],  
> > total [10.6s]/[2.1m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools  
> > {[Code Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > [124.9mb]-\>[309.7kb]/[133.1mb]}{[Par Survivor Space]  
> > [7.1mb]-\>[7.9mb]/[16.6mb]}{[CMS Old Gen]  
> > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > [2012-07-13 10:49:05,877][WARN  
> > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > [gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total  
> > [3m]/[5.2m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools {[Code Cache]  
> > [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > [95.9mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > [15.7mb]-\>[12mb]/[16.6mb]}{[CMS Old Gen]  
> > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]-\>[50.9mb]/[82mb]}  
> > r7n4.  
> > r7n5.  
> > r7n6.  
> > r7n7.  
> > r7n8.  
> > r7n9.  
> > r7n10.  
> > r7n11.  
> > r7n12.  
> > r7n13.  
> > r7n14.  
> > r7n15.  
> > r7n16.  
> > r7n17.  
> > r7n18.  
> > [2012-07-13 10:32:32,003][WARN  
> > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > [gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total  
> > [4.1s]/[9.2s], memory [1.4gb]-\>[1.3gb]/[15.9gb], all\_pools {[Code  
> > Cache] [10.5mb]-\>[10.5mb]/[48mb]}{[Par Eden Space]  
> > [59.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > [10.1mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > [1.3gb]-\>[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]-\>[56.3mb]/[82mb]}  
> > [2012-07-13 10:37:14,787][WARN  
> > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > [gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total  
> > [2.7m]/[2.9m], memory [1.5gb]-\>[1.5gb]/[15.9gb], all\_pools {[Code  
> > Cache] [10.6mb]-\>[10.6mb]/[48mb]}{[Par Eden Space]  
> > [1.3mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > [6.6mb]-\>[13.3mb]/[16.6mb]}{[CMS Old Gen]  
> > [1.5gb]-\>[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]-\>[56.7mb]/[82mb]}  
> > r7n19.  
> > r7n20.

---

<div class="post-metadata">

**Author:** ![Filirom1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/filirom1/32/2234_2.png) [@Filirom1](https://discuss.elastic.co/u/Filirom1)\
**Post date:** [July 16, 2012, 1:41pm UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/4 "2012-07-16T13:41:23Z")

</div>

It may be because of the leap second:  
[http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/](http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/)

Try this:  
[https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo)

Romain

2012/7/16 Thibaut Britz [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)

> Hi Shay,
> 
> Swap is turned off and no swap partition exists, bootstrap.mlockall is  
> enabled. They are all dedicated servers running on ubuntu 11.04, java  
> "1.6.0\_24"  
> We will activate the extended GC logging and I will post the results.
> 
> The problem seems to mostly occur when we restart the cluster. So I  
> suspect it to to be related to a relatively high indexation volume  
> when the cluster is newly started.
> 
> Thanks,  
> Thibaut
> 
> On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> 
> > You might simply had it before as well, except that in 0.19, this is  
> > logged  
> > now (the logging part is new). Such long GC pauses for the Par New  
> > generation are strange, please check if you don't end up swapping the JVM  
> > (and also turn on bootstrap.mlockall, check the installation guide).
> > 
> > Also, where are you running elasticsearch on? Is it AWS? If so, which OS  
> > are  
> > you using?
> > 
> > Last, a bit more details can be obtained by turning on GC logging  
> > information. You can head to bin/elasticsearch.in.sh, and uncomment the  
> > -XX:+PrintGCDetails and the path to the GC logging file. What would be  
> > interesting there is to see the details associated with such long  
> > collections (and see if its actually user time being spent doing them).
> > 
> > On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz  
> > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote:
> > 
> > > Hi,
> > > 
> > > we just upgraded our production cluster from 0.17 to 0.18 to 0.19.  
> > > The new zone features are really great, but we observed that nodes  
> > > were constantly marked as invalid and drop out.
> > > 
> > > As it turns out, the GC sometimes needs more than 2 or 3 minutes to  
> > > process. We increased the timeout so nodes don't drop out anymore. But  
> > > still all requests to taht node will hang for a few minutes
> > > 
> > > I saw similar posts here in the last 2 weeks. Any idea what might cause  
> > > this?  
> > > We activated mlock (it would be good if there would a message in the  
> > > log saying that it's active) and used the default GC settings, but it  
> > > didn't help.
> > > 
> > > We didn't have any GC collection issues in the old version. Any ideas  
> > > what might cause this? We did however optimised the indexing process  
> > > so it's much faster. Could this be related to many documents being  
> > > indexed at the same time?
> > > 
> > > Below the GC log of our servers:  
> > > Thanks,  
> > > Thibaut
> > > 
> > > r6n1.  
> > > [2012-07-13 12:00:11,642][INFO  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],  
> > > total [787ms]/[7.8m], memory [5gb]-\>[5gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [49.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [13.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]-\>[5gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:00:26,991][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],  
> > > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [12mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [14.8mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:00:29,288][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],  
> > > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [1.2mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.7mb]-\>[7.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:00:48,606][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],  
> > > total [7.9s]/[8.1m], memory [5.4gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [126mb]-\>[364kb]/[133.1mb]}{[Par Survivor Space]  
> > > [7.2mb]-\>[10.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:01:12,728][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],  
> > > total [21.9s]/[8.4m], memory [5.4gb]-\>[5.4gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [2.5mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [14.3mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.4gb]-\>[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:01:38,008][INFO  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],  
> > > total [2.5s]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [68.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [4.8mb]-\>[10mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:01:45,613][INFO  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],  
> > > total [702ms]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [100.3mb]-\>[4.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > [8.4mb]-\>[7.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:02:25,657][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],  
> > > total [23.3s]/[9m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [77.7mb]-\>[4.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.7mb]-\>[6.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:04:43,299][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],  
> > > total [2.2m]/[11.2m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [4.1mb]-\>[747.9kb]/[133.1mb]}{[Par Survivor Space]  
> > > [6.8mb]-\>[16.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > [2012-07-13 12:04:59,157][WARN  
> > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > [gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],  
> > > total [15.4s]/[11.5m], memory [5.9gb]-\>[6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > [747.9kb]-\>[388.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > [16.5mb]-\>[16.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > r6n2.  
> > > r6n3.  
> > > r6n4.  
> > > r6n5.  
> > > r6n6.  
> > > [2012-07-13 13:10:26,899][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],  
> > > total [20.8s]/[14.1m], memory [8.7gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [70.7mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [11.9mb]-\>[8.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:10:35,548][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],  
> > > total [5.8s]/[14.2m], memory [8.6gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [18.7kb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.4mb]-\>[5.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:10:41,899][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],  
> > > total [4.1s]/[14.3m], memory [8.8gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [110.6mb]-\>[115.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > [6.1mb]-\>[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:11:14,189][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],  
> > > total [8.3s]/[14.5m], memory [8.7gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [439.9kb]-\>[15.2kb]/[133.1mb]}{[Par Survivor Space]  
> > > [6.9mb]-\>[7.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:11:50,448][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],  
> > > total [30.7s]/[15m], memory [8.8gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [57.1mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.8mb]-\>[2.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:12:05,445][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],  
> > > total [10.1s]/[15.2m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [2.4mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [558.9kb]-\>[300.6kb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:13:25,340][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],  
> > > total [1.2m]/[16.4m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [25.5mb]-\>[126.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > [3.9mb]-\>[3.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > [2012-07-13 13:14:56,007][WARN  
> > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > [gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],  
> > > total [1.2m]/[17.8m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > [78.8mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > [6.3mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > r6n7.  
> > > r6n8.  
> > > [2012-07-13 12:43:21,850][WARN  
> > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > [gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],  
> > > total [1.1m]/[4.6m], memory [3.9gb]-\>[3.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [33.6mb]-\>[2.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [11mb]-\>[9.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > [3.8gb]-\>[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > [2012-07-13 12:43:35,199][WARN  
> > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > [gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total  
> > > [3s]/[4.7m], memory [3.9gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> > > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [77.9mb]-\>[1.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > [12.9mb]-\>[9mb]/[16.6mb]}{[CMS Old Gen]  
> > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > [2012-07-13 12:44:02,539][WARN  
> > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > [gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],  
> > > total [6.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [73.3mb]-\>[1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [8.2mb]-\>[8.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > [2012-07-13 12:44:40,213][WARN  
> > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > [gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total  
> > > [2.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> > > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [124.3mb]-\>[177.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > [9.9mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > r6n9.  
> > > r6n10.  
> > > r6n11.  
> > > r6n12.  
> > > r6n13.  
> > > r6n14.  
> > > r6n15.  
> > > r6n16.  
> > > r6n17.  
> > > r6n18.  
> > > r6n19.  
> > > r6n20.  
> > > r7n1.  
> > > [2012-07-13 11:01:45,714][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],  
> > > total [4.2s]/[1m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [65mb]-\>[291.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.7mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 11:01:50,902][INFO  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],  
> > > total [4.2s]/[1m], memory [5.8gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [291.8kb]-\>[8.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.2mb]-\>[2.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 11:02:00,444][INFO  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],  
> > > total [806ms]/[1.1m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [119mb]-\>[74.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [915.6kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 11:02:26,492][INFO  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],  
> > > total [3.7s]/[1.1m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [776.8kb]-\>[969.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > [449.2kb]-\>[1.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 11:02:36,529][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],  
> > > total [2.8s]/[1.2m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [125mb]-\>[2.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 11:02:41,545][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total  
> > > [4.3s]/[1.3m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [2.3mb]-\>[5mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 11:04:57,122][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],  
> > > total [4.4s]/[1.4m], memory [6.2gb]-\>[6.1gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > [78.5mb]-\>[616.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.7mb]-\>[925.9kb]/[16.6mb]}{[CMS Old Gen]  
> > > [6.1gb]-\>[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > [2012-07-13 12:21:35,881][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total  
> > > [1.8s]/[3.5m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [129.3mb]-\>[3.4mb]/[133.1mb]}{[Par Survivor Space]  
> > > [10.4mb]-\>[9.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:21:46,480][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],  
> > > total [4.8s]/[3.6m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [39.7mb]-\>[10.5mb]/[133.1mb]}{[Par Survivor Space]  
> > > [4.1mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:22:11,706][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],  
> > > total [1.9s]/[3.7m], memory [6.9gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [27.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [8.3mb]-\>[6.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:22:38,482][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],  
> > > total [2.1s]/[3.7m], memory [7gb]-\>[6.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [55.2mb]-\>[295.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > [9.8mb]-\>[15.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:22:59,114][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],  
> > > total [3.8s]/[3.8m], memory [7.1gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [131.2mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.6mb]-\>[2.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:23:11,637][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total  
> > > [1s]/[3.9m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [63.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1mb]-\>[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:23:44,223][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total  
> > > [1.7s]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [1.1mb]-\>[16.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.4mb]-\>[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:23:49,542][INFO  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],  
> > > total [724ms]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [2.5mb]-\>[2.7mb]/[133.1mb]}{[Par Survivor Space]  
> > > [913.9kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:24:48,348][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],  
> > > total [2.2s]/[4.3m], memory [7.1gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [1.2mb]-\>[1.6mb]/[133.1mb]}{[Par Survivor Space]  
> > > [736.2kb]-\>[783.1kb]/[16.6mb]}{[CMS Old Gen]  
> > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:24:52,591][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total  
> > > [1s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools {[Code Cache]  
> > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [92.1mb]-\>[67.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > [899.7kb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:24:58,726][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],  
> > > total [3.2s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [120.9mb]-\>[9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [5.1mb]-\>[4.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:25:19,635][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],  
> > > total [6.8s]/[4.5m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [105.4mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.7mb]-\>[1.9mb]/[16.6mb]}{[CMS Old Gen]  
> > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:25:46,737][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],  
> > > total [5.2s]/[4.6m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [115.8mb]-\>[12.5kb]/[133.1mb]}{[Par Survivor Space]  
> > > [862kb]-\>[548.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 12:25:53,811][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],  
> > > total [4.7s]/[4.7m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > [120mb]-\>[2.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [689.2kb]-\>[680.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > [2012-07-13 13:07:55,134][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],  
> > > total [5.6s]/[5.6m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [99.4mb]-\>[357.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > [6.7mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:08:06,093][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],  
> > > total [2.3s]/[5.6m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [34.4mb]-\>[572.1kb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.3mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:08:31,880][INFO  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],  
> > > total [3.6s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [26.5mb]-\>[483.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > [3mb]-\>[797.2kb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:08:35,299][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],  
> > > total [3.2s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [483.4kb]-\>[551.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > [797.2kb]-\>[5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:08:41,541][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],  
> > > total [2.4s]/[5.8m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [6mb]-\>[28.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > [300.9kb]-\>[374.1kb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:08:48,737][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],  
> > > total [3.3s]/[5.8m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [68.2mb]-\>[50.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.7mb]-\>[1.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:08:55,241][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],  
> > > total [1.1s]/[5.9m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [52.9kb]-\>[16.5kb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.4mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:09:24,266][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],  
> > > total [1.9s]/[5.9m], memory [5.7gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [1.1mb]-\>[206.9kb]/[133.1mb]}{[Par Survivor Space]  
> > > [3.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:09:59,338][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],  
> > > total [9.1s]/[6.1m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [97.6mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.1mb]-\>[1.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:10:07,579][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],  
> > > total [3.3s]/[6.2m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [58.9mb]-\>[5.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.1mb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS  
> > > Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:10:30,327][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],  
> > > total [6.6s]/[6.3m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [44.2mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [633.6kb]-\>[685.6kb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:10:40,008][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],  
> > > total [1.9s]/[6.4m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [42.9mb]-\>[174.3kb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.1mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:11:22,592][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],  
> > > total [1.5s]/[6.4m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [50.5mb]-\>[3.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > [2.9mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:11:35,865][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],  
> > > total [10.8s]/[6.6m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [37.6mb]-\>[7.5mb]/[133.1mb]}{[Par Survivor Space]  
> > > [181.5kb]-\>[133.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:11:37,723][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],  
> > > total [1.3s]/[6.6m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [7.5mb]-\>[18.3kb]/[133.1mb]}{[Par Survivor Space]  
> > > [133.3kb]-\>[540kb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:11:47,016][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],  
> > > total [3s]/[6.7m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [63.3mb]-\>[3.6mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.5mb]-\>[1.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:11:55,906][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],  
> > > total [2.3s]/[6.7m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [126.9mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [1.5mb]-\>[1.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > [2012-07-13 13:12:23,075][WARN  
> > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > [gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],  
> > > total [1.1s]/[6.8m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > [1.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [184.2kb]-\>[237.9kb]/[16.6mb]}{[CMS Old Gen]  
> > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > r7n2.  
> > > r7n3.  
> > > [2012-07-13 10:44:24,127][WARN  
> > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > [gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],  
> > > total [10.2s]/[1m], memory [8.9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > [108.3mb]-\>[24mb]/[133.1mb]}{[Par Survivor Space]  
> > > [10.9mb]-\>[7.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > [2012-07-13 10:45:02,249][WARN  
> > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > [gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],  
> > > total [36.5s]/[1.6m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > [95mb]-\>[9.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > [11.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > [2012-07-13 10:45:43,819][WARN  
> > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > [gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],  
> > > total [15.1s]/[1.9m], memory [9.4gb]-\>[9.3gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [12.9mb]-\>[12.9mb]/[48mb]}{[Par Eden Space]  
> > > [121.2mb]-\>[13.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [7.9mb]-\>[15.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [9.3gb]-\>[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > [2012-07-13 10:46:01,557][WARN  
> > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > [gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],  
> > > total [10.6s]/[2.1m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools  
> > > {[Code Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > [124.9mb]-\>[309.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > [7.1mb]-\>[7.9mb]/[16.6mb]}{[CMS Old Gen]  
> > > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > [2012-07-13 10:49:05,877][WARN  
> > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > [gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total  
> > > [3m]/[5.2m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools {[Code Cache]  
> > > [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > [95.9mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > [15.7mb]-\>[12mb]/[16.6mb]}{[CMS Old Gen]  
> > > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]-\>[50.9mb]/[82mb]}  
> > > r7n4.  
> > > r7n5.  
> > > r7n6.  
> > > r7n7.  
> > > r7n8.  
> > > r7n9.  
> > > r7n10.  
> > > r7n11.  
> > > r7n12.  
> > > r7n13.  
> > > r7n14.  
> > > r7n15.  
> > > r7n16.  
> > > r7n17.  
> > > r7n18.  
> > > [2012-07-13 10:32:32,003][WARN  
> > > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > > [gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total  
> > > [4.1s]/[9.2s], memory [1.4gb]-\>[1.3gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [10.5mb]-\>[10.5mb]/[48mb]}{[Par Eden Space]  
> > > [59.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > [10.1mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > [1.3gb]-\>[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]-\>[56.3mb]/[82mb]}  
> > > [2012-07-13 10:37:14,787][WARN  
> > > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > > [gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total  
> > > [2.7m]/[2.9m], memory [1.5gb]-\>[1.5gb]/[15.9gb], all\_pools {[Code  
> > > Cache] [10.6mb]-\>[10.6mb]/[48mb]}{[Par Eden Space]  
> > > [1.3mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > > [6.6mb]-\>[13.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > [1.5gb]-\>[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]-\>[56.7mb]/[82mb]}  
> > > r7n19.  
> > > r7n20.

---

<div class="post-metadata">

**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\
**Post date:** [July 16, 2012, 4:38pm UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/5 "2012-07-16T16:38:33Z")

</div>

Thanks for the hint.

We were hit by the leap second bug and already restarted all our  
servers directly afterwards. Unfortunately, it's not related to that.

Thanks,  
Thibaut

On Mon, Jul 16, 2012 at 3:41 PM, Romain [filirom1@gmail.com](mailto:filirom1@gmail.com) wrote:

> It may be because of the leap second:  
> [http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/](http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/)
> 
> Try this:  
> [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo)
> 
> Romain
> 
> 2012/7/16 Thibaut Britz [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)
> 
> > Hi Shay,
> > 
> > Swap is turned off and no swap partition exists, bootstrap.mlockall is  
> > enabled. They are all dedicated servers running on ubuntu 11.04, java  
> > "1.6.0\_24"  
> > We will activate the extended GC logging and I will post the results.
> > 
> > The problem seems to mostly occur when we restart the cluster. So I  
> > suspect it to to be related to a relatively high indexation volume  
> > when the cluster is newly started.
> > 
> > Thanks,  
> > Thibaut
> > 
> > On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> > 
> > > You might simply had it before as well, except that in 0.19, this is  
> > > logged  
> > > now (the logging part is new). Such long GC pauses for the Par New  
> > > generation are strange, please check if you don't end up swapping the  
> > > JVM  
> > > (and also turn on bootstrap.mlockall, check the installation guide).
> > > 
> > > Also, where are you running elasticsearch on? Is it AWS? If so, which OS  
> > > are  
> > > you using?
> > > 
> > > Last, a bit more details can be obtained by turning on GC logging  
> > > information. You can head to bin/elasticsearch.in.sh, and uncomment the  
> > > -XX:+PrintGCDetails and the path to the GC logging file. What would be  
> > > interesting there is to see the details associated with such long  
> > > collections (and see if its actually user time being spent doing them).
> > > 
> > > On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz  
> > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote:
> > > 
> > > > Hi,
> > > > 
> > > > we just upgraded our production cluster from 0.17 to 0.18 to 0.19.  
> > > > The new zone features are really great, but we observed that nodes  
> > > > were constantly marked as invalid and drop out.
> > > > 
> > > > As it turns out, the GC sometimes needs more than 2 or 3 minutes to  
> > > > process. We increased the timeout so nodes don't drop out anymore. But  
> > > > still all requests to taht node will hang for a few minutes
> > > > 
> > > > I saw similar posts here in the last 2 weeks. Any idea what might cause  
> > > > this?  
> > > > We activated mlock (it would be good if there would a message in the  
> > > > log saying that it's active) and used the default GC settings, but it  
> > > > didn't help.
> > > > 
> > > > We didn't have any GC collection issues in the old version. Any ideas  
> > > > what might cause this? We did however optimised the indexing process  
> > > > so it's much faster. Could this be related to many documents being  
> > > > indexed at the same time?
> > > > 
> > > > Below the GC log of our servers:  
> > > > Thanks,  
> > > > Thibaut
> > > > 
> > > > r6n1.  
> > > > [2012-07-13 12:00:11,642][INFO  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],  
> > > > total [787ms]/[7.8m], memory [5gb]-\>[5gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [49.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [13.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]-\>[5gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:00:26,991][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],  
> > > > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [12mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [14.8mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:00:29,288][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],  
> > > > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [1.2mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.7mb]-\>[7.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:00:48,606][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],  
> > > > total [7.9s]/[8.1m], memory [5.4gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [126mb]-\>[364kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [7.2mb]-\>[10.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:01:12,728][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],  
> > > > total [21.9s]/[8.4m], memory [5.4gb]-\>[5.4gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [2.5mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [14.3mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.4gb]-\>[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:01:38,008][INFO  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],  
> > > > total [2.5s]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [68.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [4.8mb]-\>[10mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:01:45,613][INFO  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],  
> > > > total [702ms]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [100.3mb]-\>[4.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [8.4mb]-\>[7.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:02:25,657][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],  
> > > > total [23.3s]/[9m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [77.7mb]-\>[4.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.7mb]-\>[6.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:04:43,299][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],  
> > > > total [2.2m]/[11.2m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [4.1mb]-\>[747.9kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [6.8mb]-\>[16.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > [2012-07-13 12:04:59,157][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > [gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],  
> > > > total [15.4s]/[11.5m], memory [5.9gb]-\>[6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > [747.9kb]-\>[388.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [16.5mb]-\>[16.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > r6n2.  
> > > > r6n3.  
> > > > r6n4.  
> > > > r6n5.  
> > > > r6n6.  
> > > > [2012-07-13 13:10:26,899][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],  
> > > > total [20.8s]/[14.1m], memory [8.7gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [70.7mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [11.9mb]-\>[8.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:10:35,548][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],  
> > > > total [5.8s]/[14.2m], memory [8.6gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [18.7kb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.4mb]-\>[5.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:10:41,899][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],  
> > > > total [4.1s]/[14.3m], memory [8.8gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [110.6mb]-\>[115.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [6.1mb]-\>[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:11:14,189][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],  
> > > > total [8.3s]/[14.5m], memory [8.7gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [439.9kb]-\>[15.2kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [6.9mb]-\>[7.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:11:50,448][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],  
> > > > total [30.7s]/[15m], memory [8.8gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [57.1mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.8mb]-\>[2.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:12:05,445][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],  
> > > > total [10.1s]/[15.2m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [2.4mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [558.9kb]-\>[300.6kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:13:25,340][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],  
> > > > total [1.2m]/[16.4m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [25.5mb]-\>[126.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [3.9mb]-\>[3.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > [2012-07-13 13:14:56,007][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > [gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],  
> > > > total [1.2m]/[17.8m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > [78.8mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [6.3mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > r6n7.  
> > > > r6n8.  
> > > > [2012-07-13 12:43:21,850][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > [gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],  
> > > > total [1.1m]/[4.6m], memory [3.9gb]-\>[3.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [33.6mb]-\>[2.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [11mb]-\>[9.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [3.8gb]-\>[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > [2012-07-13 12:43:35,199][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > [gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total  
> > > > [3s]/[4.7m], memory [3.9gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [77.9mb]-\>[1.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [12.9mb]-\>[9mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > [2012-07-13 12:44:02,539][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > [gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],  
> > > > total [6.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [73.3mb]-\>[1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [8.2mb]-\>[8.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > [2012-07-13 12:44:40,213][WARN  
> > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > [gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total  
> > > > [2.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [124.3mb]-\>[177.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [9.9mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > r6n9.  
> > > > r6n10.  
> > > > r6n11.  
> > > > r6n12.  
> > > > r6n13.  
> > > > r6n14.  
> > > > r6n15.  
> > > > r6n16.  
> > > > r6n17.  
> > > > r6n18.  
> > > > r6n19.  
> > > > r6n20.  
> > > > r7n1.  
> > > > [2012-07-13 11:01:45,714][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],  
> > > > total [4.2s]/[1m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [65mb]-\>[291.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.7mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 11:01:50,902][INFO  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],  
> > > > total [4.2s]/[1m], memory [5.8gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [291.8kb]-\>[8.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.2mb]-\>[2.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 11:02:00,444][INFO  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],  
> > > > total [806ms]/[1.1m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [119mb]-\>[74.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [915.6kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 11:02:26,492][INFO  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],  
> > > > total [3.7s]/[1.1m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [776.8kb]-\>[969.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [449.2kb]-\>[1.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 11:02:36,529][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],  
> > > > total [2.8s]/[1.2m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [125mb]-\>[2.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 11:02:41,545][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total  
> > > > [4.3s]/[1.3m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [2.3mb]-\>[5mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 11:04:57,122][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],  
> > > > total [4.4s]/[1.4m], memory [6.2gb]-\>[6.1gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > [78.5mb]-\>[616.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.7mb]-\>[925.9kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [6.1gb]-\>[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > [2012-07-13 12:21:35,881][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total  
> > > > [1.8s]/[3.5m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [129.3mb]-\>[3.4mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [10.4mb]-\>[9.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:21:46,480][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],  
> > > > total [4.8s]/[3.6m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [39.7mb]-\>[10.5mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [4.1mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:22:11,706][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],  
> > > > total [1.9s]/[3.7m], memory [6.9gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [27.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [8.3mb]-\>[6.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:22:38,482][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],  
> > > > total [2.1s]/[3.7m], memory [7gb]-\>[6.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [55.2mb]-\>[295.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [9.8mb]-\>[15.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:22:59,114][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],  
> > > > total [3.8s]/[3.8m], memory [7.1gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [131.2mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.6mb]-\>[2.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:23:11,637][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total  
> > > > [1s]/[3.9m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [63.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1mb]-\>[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:23:44,223][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total  
> > > > [1.7s]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [1.1mb]-\>[16.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.4mb]-\>[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:23:49,542][INFO  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],  
> > > > total [724ms]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [2.5mb]-\>[2.7mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [913.9kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:24:48,348][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],  
> > > > total [2.2s]/[4.3m], memory [7.1gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [1.2mb]-\>[1.6mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [736.2kb]-\>[783.1kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:24:52,591][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total  
> > > > [1s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [92.1mb]-\>[67.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [899.7kb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:24:58,726][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],  
> > > > total [3.2s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [120.9mb]-\>[9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [5.1mb]-\>[4.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:25:19,635][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],  
> > > > total [6.8s]/[4.5m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [105.4mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.7mb]-\>[1.9mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:25:46,737][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],  
> > > > total [5.2s]/[4.6m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [115.8mb]-\>[12.5kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [862kb]-\>[548.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 12:25:53,811][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],  
> > > > total [4.7s]/[4.7m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > [120mb]-\>[2.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [689.2kb]-\>[680.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > [2012-07-13 13:07:55,134][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],  
> > > > total [5.6s]/[5.6m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [99.4mb]-\>[357.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [6.7mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:08:06,093][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],  
> > > > total [2.3s]/[5.6m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [34.4mb]-\>[572.1kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.3mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:08:31,880][INFO  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],  
> > > > total [3.6s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [26.5mb]-\>[483.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [3mb]-\>[797.2kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:08:35,299][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],  
> > > > total [3.2s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [483.4kb]-\>[551.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [797.2kb]-\>[5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:08:41,541][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],  
> > > > total [2.4s]/[5.8m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [6mb]-\>[28.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [300.9kb]-\>[374.1kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:08:48,737][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],  
> > > > total [3.3s]/[5.8m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [68.2mb]-\>[50.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.7mb]-\>[1.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:08:55,241][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],  
> > > > total [1.1s]/[5.9m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [52.9kb]-\>[16.5kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.4mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:09:24,266][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],  
> > > > total [1.9s]/[5.9m], memory [5.7gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [1.1mb]-\>[206.9kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [3.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:09:59,338][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],  
> > > > total [9.1s]/[6.1m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [97.6mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.1mb]-\>[1.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:10:07,579][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],  
> > > > total [3.3s]/[6.2m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [58.9mb]-\>[5.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.1mb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS  
> > > > Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:10:30,327][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],  
> > > > total [6.6s]/[6.3m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [44.2mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [633.6kb]-\>[685.6kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:10:40,008][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],  
> > > > total [1.9s]/[6.4m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [42.9mb]-\>[174.3kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.1mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:11:22,592][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],  
> > > > total [1.5s]/[6.4m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [50.5mb]-\>[3.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [2.9mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:11:35,865][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],  
> > > > total [10.8s]/[6.6m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [37.6mb]-\>[7.5mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [181.5kb]-\>[133.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:11:37,723][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],  
> > > > total [1.3s]/[6.6m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [7.5mb]-\>[18.3kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [133.3kb]-\>[540kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:11:47,016][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],  
> > > > total [3s]/[6.7m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [63.3mb]-\>[3.6mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.5mb]-\>[1.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:11:55,906][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],  
> > > > total [2.3s]/[6.7m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [126.9mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [1.5mb]-\>[1.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > [2012-07-13 13:12:23,075][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > [gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],  
> > > > total [1.1s]/[6.8m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > [1.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [184.2kb]-\>[237.9kb]/[16.6mb]}{[CMS Old Gen]  
> > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > r7n2.  
> > > > r7n3.  
> > > > [2012-07-13 10:44:24,127][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > [gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],  
> > > > total [10.2s]/[1m], memory [8.9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > [108.3mb]-\>[24mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [10.9mb]-\>[7.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > [2012-07-13 10:45:02,249][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > [gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],  
> > > > total [36.5s]/[1.6m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > [95mb]-\>[9.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [11.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > [2012-07-13 10:45:43,819][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > [gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],  
> > > > total [15.1s]/[1.9m], memory [9.4gb]-\>[9.3gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [12.9mb]-\>[12.9mb]/[48mb]}{[Par Eden Space]  
> > > > [121.2mb]-\>[13.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [7.9mb]-\>[15.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [9.3gb]-\>[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > [2012-07-13 10:46:01,557][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > [gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],  
> > > > total [10.6s]/[2.1m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools  
> > > > {[Code Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > [124.9mb]-\>[309.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > [7.1mb]-\>[7.9mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > [2012-07-13 10:49:05,877][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > [gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total  
> > > > [3m]/[5.2m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > [95.9mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [15.7mb]-\>[12mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]-\>[50.9mb]/[82mb]}  
> > > > r7n4.  
> > > > r7n5.  
> > > > r7n6.  
> > > > r7n7.  
> > > > r7n8.  
> > > > r7n9.  
> > > > r7n10.  
> > > > r7n11.  
> > > > r7n12.  
> > > > r7n13.  
> > > > r7n14.  
> > > > r7n15.  
> > > > r7n16.  
> > > > r7n17.  
> > > > r7n18.  
> > > > [2012-07-13 10:32:32,003][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > > > [gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total  
> > > > [4.1s]/[9.2s], memory [1.4gb]-\>[1.3gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [10.5mb]-\>[10.5mb]/[48mb]}{[Par Eden Space]  
> > > > [59.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [10.1mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [1.3gb]-\>[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]-\>[56.3mb]/[82mb]}  
> > > > [2012-07-13 10:37:14,787][WARN  
> > > > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > > > [gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total  
> > > > [2.7m]/[2.9m], memory [1.5gb]-\>[1.5gb]/[15.9gb], all\_pools {[Code  
> > > > Cache] [10.6mb]-\>[10.6mb]/[48mb]}{[Par Eden Space]  
> > > > [1.3mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > [6.6mb]-\>[13.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > [1.5gb]-\>[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]-\>[56.7mb]/[82mb]}  
> > > > r7n19.  
> > > > r7n20.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 18, 2012, 8:46pm UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/6 "2012-07-18T20:46:16Z")

</div>

Can you try and use a newer JVM, 1.6.0\_24 is pretty old.

On Mon, Jul 16, 2012 at 9:38 AM, Thibaut Britz \<  
[thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote:

> Thanks for the hint.
> 
> We were hit by the leap second bug and already restarted all our  
> servers directly afterwards. Unfortunately, it's not related to that.
> 
> Thanks,  
> Thibaut
> 
> On Mon, Jul 16, 2012 at 3:41 PM, Romain [filirom1@gmail.com](mailto:filirom1@gmail.com) wrote:
> 
> > It may be because of the leap second:
> 
> [http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/](http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/)
> 
> > Try this:
> 
> [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo)
> 
> > Romain
> > 
> > 2012/7/16 Thibaut Britz [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)
> > 
> > > Hi Shay,
> > > 
> > > Swap is turned off and no swap partition exists, bootstrap.mlockall is  
> > > enabled. They are all dedicated servers running on ubuntu 11.04, java  
> > > "1.6.0\_24"  
> > > We will activate the extended GC logging and I will post the results.
> > > 
> > > The problem seems to mostly occur when we restart the cluster. So I  
> > > suspect it to to be related to a relatively high indexation volume  
> > > when the cluster is newly started.
> > > 
> > > Thanks,  
> > > Thibaut
> > > 
> > > On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> > > 
> > > > You might simply had it before as well, except that in 0.19, this is  
> > > > logged  
> > > > now (the logging part is new). Such long GC pauses for the Par New  
> > > > generation are strange, please check if you don't end up swapping the  
> > > > JVM  
> > > > (and also turn on bootstrap.mlockall, check the installation guide).
> > > > 
> > > > Also, where are you running elasticsearch on? Is it AWS? If so, which  
> > > > OS  
> > > > are  
> > > > you using?
> > > > 
> > > > Last, a bit more details can be obtained by turning on GC logging  
> > > > information. You can head to bin/elasticsearch.in.sh, and uncomment  
> > > > the  
> > > > -XX:+PrintGCDetails and the path to the GC logging file. What would be  
> > > > interesting there is to see the details associated with such long  
> > > > collections (and see if its actually user time being spent doing  
> > > > them).
> > > > 
> > > > On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz  
> > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > we just upgraded our production cluster from 0.17 to 0.18 to 0.19.  
> > > > > The new zone features are really great, but we observed that nodes  
> > > > > were constantly marked as invalid and drop out.
> > > > > 
> > > > > As it turns out, the GC sometimes needs more than 2 or 3 minutes to  
> > > > > process. We increased the timeout so nodes don't drop out anymore.  
> > > > > But  
> > > > > still all requests to taht node will hang for a few minutes
> > > > > 
> > > > > I saw similar posts here in the last 2 weeks. Any idea what might  
> > > > > cause  
> > > > > this?  
> > > > > We activated mlock (it would be good if there would a message in the  
> > > > > log saying that it's active) and used the default GC settings, but it  
> > > > > didn't help.
> > > > > 
> > > > > We didn't have any GC collection issues in the old version. Any ideas  
> > > > > what might cause this? We did however optimised the indexing process  
> > > > > so it's much faster. Could this be related to many documents being  
> > > > > indexed at the same time?
> > > > > 
> > > > > Below the GC log of our servers:  
> > > > > Thanks,  
> > > > > Thibaut
> > > > > 
> > > > > r6n1.  
> > > > > [2012-07-13 12:00:11,642][INFO  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],  
> > > > > total [787ms]/[7.8m], memory [5gb]-\>[5gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [49.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [13.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]-\>[5gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:00:26,991][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],  
> > > > > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [12mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [14.8mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:00:29,288][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],  
> > > > > total [2.1s]/[7.9m], memory [5.2gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [1.2mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.7mb]-\>[7.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:00:48,606][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],  
> > > > > total [7.9s]/[8.1m], memory [5.4gb]-\>[5.2gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [126mb]-\>[364kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [7.2mb]-\>[10.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.2gb]-\>[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:01:12,728][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],  
> > > > > total [21.9s]/[8.4m], memory [5.4gb]-\>[5.4gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [2.5mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [14.3mb]-\>[2.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.4gb]-\>[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:01:38,008][INFO  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],  
> > > > > total [2.5s]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [68.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [4.8mb]-\>[10mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:01:45,613][INFO  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],  
> > > > > total [702ms]/[8.5m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [100.3mb]-\>[4.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [8.4mb]-\>[7.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:02:25,657][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],  
> > > > > total [23.3s]/[9m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > > {[Code  
> > > > > Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [77.7mb]-\>[4.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.7mb]-\>[6.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:04:43,299][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],  
> > > > > total [2.2m]/[11.2m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [4.1mb]-\>[747.9kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [6.8mb]-\>[16.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > [2012-07-13 12:04:59,157][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]  
> > > > > [gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],  
> > > > > total [15.4s]/[11.5m], memory [5.9gb]-\>[6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [18.8mb]-\>[18.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [747.9kb]-\>[388.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [16.5mb]-\>[16.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]-\>[60mb]/[82mb]}  
> > > > > r6n2.  
> > > > > r6n3.  
> > > > > r6n4.  
> > > > > r6n5.  
> > > > > r6n6.  
> > > > > [2012-07-13 13:10:26,899][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],  
> > > > > total [20.8s]/[14.1m], memory [8.7gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [70.7mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [11.9mb]-\>[8.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:10:35,548][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],  
> > > > > total [5.8s]/[14.2m], memory [8.6gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [18.7kb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.4mb]-\>[5.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:10:41,899][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],  
> > > > > total [4.1s]/[14.3m], memory [8.8gb]-\>[8.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [110.6mb]-\>[115.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [6.1mb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.6gb]-\>[8.6gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:11:14,189][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],  
> > > > > total [8.3s]/[14.5m], memory [8.7gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [439.9kb]-\>[15.2kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [6.9mb]-\>[7.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:11:50,448][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],  
> > > > > total [30.7s]/[15m], memory [8.8gb]-\>[8.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [57.1mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.8mb]-\>[2.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.7gb]-\>[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:12:05,445][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],  
> > > > > total [10.1s]/[15.2m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [2.4mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [558.9kb]-\>[300.6kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:13:25,340][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],  
> > > > > total [1.2m]/[16.4m], memory [8.8gb]-\>[8.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [25.5mb]-\>[126.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [3.9mb]-\>[3.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > [2012-07-13 13:14:56,007][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]  
> > > > > [gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],  
> > > > > total [1.2m]/[17.8m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools  
> > > > > {[Code  
> > > > > Cache] [10.8mb]-\>[10.8mb]/[48mb]}{[Par Eden Space]  
> > > > > [78.8mb]-\>[1.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [6.3mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]-\>[43.1mb]/[82mb]}  
> > > > > r6n7.  
> > > > > r6n8.  
> > > > > [2012-07-13 12:43:21,850][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > > [gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],  
> > > > > total [1.1m]/[4.6m], memory [3.9gb]-\>[3.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [33.6mb]-\>[2.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [11mb]-\>[9.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [3.8gb]-\>[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > > [2012-07-13 12:43:35,199][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > > [gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total  
> > > > > [3s]/[4.7m], memory [3.9gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache]  
> > > > > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [77.9mb]-\>[1.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [12.9mb]-\>[9mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > > [2012-07-13 12:44:02,539][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > > [gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],  
> > > > > total [6.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [73.3mb]-\>[1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [8.2mb]-\>[8.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > > [2012-07-13 12:44:40,213][WARN  
> > > > > ][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]  
> > > > > [gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s],  
> > > > > total  
> > > > > [2.4s]/[4.8m], memory [4gb]-\>[3.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache]  
> > > > > [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [124.3mb]-\>[177.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [9.9mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [3.9gb]-\>[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]-\>[50mb]/[82mb]}  
> > > > > r6n9.  
> > > > > r6n10.  
> > > > > r6n11.  
> > > > > r6n12.  
> > > > > r6n13.  
> > > > > r6n14.  
> > > > > r6n15.  
> > > > > r6n16.  
> > > > > r6n17.  
> > > > > r6n18.  
> > > > > r6n19.  
> > > > > r6n20.  
> > > > > r7n1.  
> > > > > [2012-07-13 11:01:45,714][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],  
> > > > > total [4.2s]/[1m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [65mb]-\>[291.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.7mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 11:01:50,902][INFO  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],  
> > > > > total [4.2s]/[1m], memory [5.8gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [291.8kb]-\>[8.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.2mb]-\>[2.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 11:02:00,444][INFO  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],  
> > > > > total [806ms]/[1.1m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > > {[Code  
> > > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [119mb]-\>[74.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [915.6kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 11:02:26,492][INFO  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],  
> > > > > total [3.7s]/[1.1m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [776.8kb]-\>[969.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [449.2kb]-\>[1.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 11:02:36,529][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],  
> > > > > total [2.8s]/[1.2m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [125mb]-\>[2.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2mb]-\>[2.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 11:02:41,545][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total  
> > > > > [4.3s]/[1.3m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [2.3mb]-\>[5mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 11:04:57,122][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],  
> > > > > total [4.4s]/[1.4m], memory [6.2gb]-\>[6.1gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [14.1mb]-\>[14.1mb]/[48mb]}{[Par Eden Space]  
> > > > > [78.5mb]-\>[616.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.7mb]-\>[925.9kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [6.1gb]-\>[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]-\>[58.1mb]/[82mb]}  
> > > > > [2012-07-13 12:21:35,881][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s],  
> > > > > total  
> > > > > [1.8s]/[3.5m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [129.3mb]-\>[3.4mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [10.4mb]-\>[9.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:21:46,480][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],  
> > > > > total [4.8s]/[3.6m], memory [6.9gb]-\>[6.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [39.7mb]-\>[10.5mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [4.1mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [6.8gb]-\>[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:22:11,706][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],  
> > > > > total [1.9s]/[3.7m], memory [6.9gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [27.8mb]-\>[3.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [8.3mb]-\>[6.7mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:22:38,482][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],  
> > > > > total [2.1s]/[3.7m], memory [7gb]-\>[6.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [55.2mb]-\>[295.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [9.8mb]-\>[15.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:22:59,114][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],  
> > > > > total [3.8s]/[3.8m], memory [7.1gb]-\>[6.9gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [131.2mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.6mb]-\>[2.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [6.9gb]-\>[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:23:11,637][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s],  
> > > > > total  
> > > > > [1s]/[3.9m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [63.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1mb]-\>[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:23:44,223][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s],  
> > > > > total  
> > > > > [1.7s]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code Cache]  
> > > > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [1.1mb]-\>[16.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.4mb]-\>[952.2kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:23:49,542][INFO  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],  
> > > > > total [724ms]/[4m], memory [7gb]-\>[7gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [2.5mb]-\>[2.7mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [913.9kb]-\>[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]-\>[7gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:24:48,348][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],  
> > > > > total [2.2s]/[4.3m], memory [7.1gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [1.2mb]-\>[1.6mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [736.2kb]-\>[783.1kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:24:52,591][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s],  
> > > > > total  
> > > > > [1s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache]  
> > > > > [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [92.1mb]-\>[67.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [899.7kb]-\>[2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:24:58,726][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],  
> > > > > total [3.2s]/[4.3m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [120.9mb]-\>[9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [5.1mb]-\>[4.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:25:19,635][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],  
> > > > > total [6.8s]/[4.5m], memory [7.2gb]-\>[7.1gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [105.4mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.7mb]-\>[1.9mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [7.1gb]-\>[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:25:46,737][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],  
> > > > > total [5.2s]/[4.6m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [115.8mb]-\>[12.5kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [862kb]-\>[548.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 12:25:53,811][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],  
> > > > > total [4.7s]/[4.7m], memory [1.8gb]-\>[1.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15mb]-\>[15mb]/[48mb]}{[Par Eden Space]  
> > > > > [120mb]-\>[2.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [689.2kb]-\>[680.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [1.7gb]-\>[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]-\>[58.3mb]/[82mb]}  
> > > > > [2012-07-13 13:07:55,134][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],  
> > > > > total [5.6s]/[5.6m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [99.4mb]-\>[357.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [6.7mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:08:06,093][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],  
> > > > > total [2.3s]/[5.6m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [34.4mb]-\>[572.1kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.3mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:08:31,880][INFO  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],  
> > > > > total [3.6s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [26.5mb]-\>[483.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [3mb]-\>[797.2kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:08:35,299][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],  
> > > > > total [3.2s]/[5.7m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [483.4kb]-\>[551.4kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [797.2kb]-\>[5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:08:41,541][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],  
> > > > > total [2.4s]/[5.8m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [6mb]-\>[28.8kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [300.9kb]-\>[374.1kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:08:48,737][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],  
> > > > > total [3.3s]/[5.8m], memory [5.7gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [68.2mb]-\>[50.6kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.7mb]-\>[1.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:08:55,241][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],  
> > > > > total [1.1s]/[5.9m], memory [5.6gb]-\>[5.6gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [52.9kb]-\>[16.5kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.4mb]-\>[3.2mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.6gb]-\>[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:09:24,266][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],  
> > > > > total [1.9s]/[5.9m], memory [5.7gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [1.1mb]-\>[206.9kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [3.2mb]-\>[2.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:09:59,338][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],  
> > > > > total [9.1s]/[6.1m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [97.6mb]-\>[1.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.1mb]-\>[1.6mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:10:07,579][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],  
> > > > > total [3.3s]/[6.2m], memory [5.8gb]-\>[5.7gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [58.9mb]-\>[5.8mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.1mb]-\>[1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.7gb]-\>[5.7gb]/[15.8gb]}{[CMS  
> > > > > Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:10:30,327][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],  
> > > > > total [6.6s]/[6.3m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [44.2mb]-\>[2.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [633.6kb]-\>[685.6kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:10:40,008][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],  
> > > > > total [1.9s]/[6.4m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [42.9mb]-\>[174.3kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.1mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:11:22,592][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],  
> > > > > total [1.5s]/[6.4m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [50.5mb]-\>[3.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [2.9mb]-\>[1.1mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:11:35,865][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],  
> > > > > total [10.8s]/[6.6m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [37.6mb]-\>[7.5mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [181.5kb]-\>[133.3kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:11:37,723][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],  
> > > > > total [1.3s]/[6.6m], memory [5.8gb]-\>[5.8gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [7.5mb]-\>[18.3kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [133.3kb]-\>[540kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:11:47,016][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],  
> > > > > total [3s]/[6.7m], memory [5.9gb]-\>[5.8gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [63.3mb]-\>[3.6mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.5mb]-\>[1.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:11:55,906][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],  
> > > > > total [2.3s]/[6.7m], memory [6gb]-\>[5.9gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [126.9mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [1.5mb]-\>[1.4mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.8gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > [2012-07-13 13:12:23,075][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]  
> > > > > [gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],  
> > > > > total [1.1s]/[6.8m], memory [5.9gb]-\>[5.9gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [15.6mb]-\>[15.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [1.1mb]-\>[1.2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [184.2kb]-\>[237.9kb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [5.9gb]-\>[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]-\>[58.4mb]/[82mb]}  
> > > > > r7n2.  
> > > > > r7n3.  
> > > > > [2012-07-13 10:44:24,127][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > > [gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],  
> > > > > total [10.2s]/[1m], memory [8.9gb]-\>[8.9gb]/[15.9gb], all\_pools  
> > > > > {[Code  
> > > > > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > > [108.3mb]-\>[24mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [10.9mb]-\>[7.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.8gb]-\>[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > > [2012-07-13 10:45:02,249][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > > [gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],  
> > > > > total [36.5s]/[1.6m], memory [9gb]-\>[8.9gb]/[15.9gb], all\_pools  
> > > > > {[Code  
> > > > > Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > > [95mb]-\>[9.3mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [11.9mb]-\>[5.8mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [8.9gb]-\>[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > > [2012-07-13 10:45:43,819][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > > [gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],  
> > > > > total [15.1s]/[1.9m], memory [9.4gb]-\>[9.3gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [12.9mb]-\>[12.9mb]/[48mb]}{[Par Eden Space]  
> > > > > [121.2mb]-\>[13.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [7.9mb]-\>[15.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [9.3gb]-\>[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > > [2012-07-13 10:46:01,557][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > > [gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],  
> > > > > total [10.6s]/[2.1m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools  
> > > > > {[Code Cache] [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > > [124.9mb]-\>[309.7kb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [7.1mb]-\>[7.9mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]-\>[50.8mb]/[82mb]}  
> > > > > [2012-07-13 10:49:05,877][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]  
> > > > > [gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total  
> > > > > [3m]/[5.2m], memory [9.5gb]-\>[9.4gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache]  
> > > > > [13mb]-\>[13mb]/[48mb]}{[Par Eden Space]  
> > > > > [95.9mb]-\>[1.1mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [15.7mb]-\>[12mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [9.4gb]-\>[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]-\>[50.9mb]/[82mb]}  
> > > > > r7n4.  
> > > > > r7n5.  
> > > > > r7n6.  
> > > > > r7n7.  
> > > > > r7n8.  
> > > > > r7n9.  
> > > > > r7n10.  
> > > > > r7n11.  
> > > > > r7n12.  
> > > > > r7n13.  
> > > > > r7n14.  
> > > > > r7n15.  
> > > > > r7n16.  
> > > > > r7n17.  
> > > > > r7n18.  
> > > > > [2012-07-13 10:32:32,003][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > > > > [gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total  
> > > > > [4.1s]/[9.2s], memory [1.4gb]-\>[1.3gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [10.5mb]-\>[10.5mb]/[48mb]}{[Par Eden Space]  
> > > > > [59.6mb]-\>[3.9mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [10.1mb]-\>[9.5mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [1.3gb]-\>[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]-\>[56.3mb]/[82mb]}  
> > > > > [2012-07-13 10:37:14,787][WARN  
> > > > > ][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]  
> > > > > [gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m],  
> > > > > total  
> > > > > [2.7m]/[2.9m], memory [1.5gb]-\>[1.5gb]/[15.9gb], all\_pools {[Code  
> > > > > Cache] [10.6mb]-\>[10.6mb]/[48mb]}{[Par Eden Space]  
> > > > > [1.3mb]-\>[2mb]/[133.1mb]}{[Par Survivor Space]  
> > > > > [6.6mb]-\>[13.3mb]/[16.6mb]}{[CMS Old Gen]  
> > > > > [1.5gb]-\>[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]-\>[56.7mb]/[82mb]}  
> > > > > r7n19.  
> > > > > r7n20.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/gc-issues-after-upgrading-to-0-19/8405/7 "2017-07-06T03:19:54Z")

</div>