GC issues after upgrading to 0.19


(Thibaut Britz) #1

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore. But
still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might cause this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.

We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?

Below the GC log of our servers:
Thanks,
Thibaut

r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]->[8.6gb]/[15.8gb]}{[CMS
Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total
[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[15.8gb]}{[CMS
Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total
[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total
[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total
[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total
[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]->[5.7gb]/[15.8gb]}{[CMS
Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code Cache]
[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total
[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.


(Shay Banon) #2

You might simply had it before as well, except that in 0.19, this is logged
now (the logging part is new). Such long GC pauses for the Par New
generation are strange, please check if you don't end up swapping the JVM
(and also turn on bootstrap.mlockall, check the installation guide).

Also, where are you running elasticsearch on? Is it AWS? If so, which OS
are you using?

Last, a bit more details can be obtained by turning on GC logging
information. You can head to bin/elasticsearch.in.sh, and uncomment
the -XX:+PrintGCDetails and the path to the GC logging file. What would be
interesting there is to see the details associated with such long
collections (and see if its actually user time being spent doing them).

On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz <
thibaut.britz@trendiction.com> wrote:

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore. But
still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might cause
this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.

We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?

Below the GC log of our servers:
Thanks,
Thibaut

r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]->[8.6gb]/[15.8gb]}{[CMS
Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total
[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[15.8gb]}{[CMS
Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total
[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total
[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total
[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total
[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]->[5.7gb]/[15.8gb]}{[CMS
Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code Cache]
[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total
[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.


(Thibaut Britz) #3

Hi Shay,

Swap is turned off and no swap partition exists, bootstrap.mlockall is
enabled. They are all dedicated servers running on ubuntu 11.04, java
"1.6.0_24"
We will activate the extended GC logging and I will post the results.

The problem seems to mostly occur when we restart the cluster. So I
suspect it to to be related to a relatively high indexation volume
when the cluster is newly started.

Thanks,
Thibaut

On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon kimchy@gmail.com wrote:

You might simply had it before as well, except that in 0.19, this is logged
now (the logging part is new). Such long GC pauses for the Par New
generation are strange, please check if you don't end up swapping the JVM
(and also turn on bootstrap.mlockall, check the installation guide).

Also, where are you running elasticsearch on? Is it AWS? If so, which OS are
you using?

Last, a bit more details can be obtained by turning on GC logging
information. You can head to bin/elasticsearch.in.sh, and uncomment the
-XX:+PrintGCDetails and the path to the GC logging file. What would be
interesting there is to see the details associated with such long
collections (and see if its actually user time being spent doing them).

On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz
thibaut.britz@trendiction.com wrote:

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore. But
still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might cause
this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.

We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?

Below the GC log of our servers:
Thanks,
Thibaut

r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]->[8.6gb]/[15.8gb]}{[CMS
Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total
[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[15.8gb]}{[CMS
Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total
[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total
[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total
[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total
[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]->[5.7gb]/[15.8gb]}{[CMS
Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code Cache]
[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total
[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.


(Filirom1) #4

It may be because of the leap second:
http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/

Try this:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo

Romain

2012/7/16 Thibaut Britz thibaut.britz@trendiction.com

Hi Shay,

Swap is turned off and no swap partition exists, bootstrap.mlockall is
enabled. They are all dedicated servers running on ubuntu 11.04, java
"1.6.0_24"
We will activate the extended GC logging and I will post the results.

The problem seems to mostly occur when we restart the cluster. So I
suspect it to to be related to a relatively high indexation volume
when the cluster is newly started.

Thanks,
Thibaut

On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon kimchy@gmail.com wrote:

You might simply had it before as well, except that in 0.19, this is
logged
now (the logging part is new). Such long GC pauses for the Par New
generation are strange, please check if you don't end up swapping the JVM
(and also turn on bootstrap.mlockall, check the installation guide).

Also, where are you running elasticsearch on? Is it AWS? If so, which OS
are
you using?

Last, a bit more details can be obtained by turning on GC logging
information. You can head to bin/elasticsearch.in.sh, and uncomment the
-XX:+PrintGCDetails and the path to the GC logging file. What would be
interesting there is to see the details associated with such long
collections (and see if its actually user time being spent doing them).

On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz
thibaut.britz@trendiction.com wrote:

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore. But
still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might cause
this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.

We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?

Below the GC log of our servers:
Thanks,
Thibaut

r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]->[8.6gb]/[15.8gb]}{[CMS
Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total
[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[15.8gb]}{[CMS
Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total
[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total
[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total
[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total
[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]->[5.7gb]/[15.8gb]}{[CMS
Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code Cache]
[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total
[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.


(Thibaut Britz) #5

Thanks for the hint.

We were hit by the leap second bug and already restarted all our
servers directly afterwards. Unfortunately, it's not related to that.

Thanks,
Thibaut

On Mon, Jul 16, 2012 at 3:41 PM, Romain filirom1@gmail.com wrote:

It may be because of the leap second:
http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/

Try this:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo

Romain

2012/7/16 Thibaut Britz thibaut.britz@trendiction.com

Hi Shay,

Swap is turned off and no swap partition exists, bootstrap.mlockall is
enabled. They are all dedicated servers running on ubuntu 11.04, java
"1.6.0_24"
We will activate the extended GC logging and I will post the results.

The problem seems to mostly occur when we restart the cluster. So I
suspect it to to be related to a relatively high indexation volume
when the cluster is newly started.

Thanks,
Thibaut

On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon kimchy@gmail.com wrote:

You might simply had it before as well, except that in 0.19, this is
logged
now (the logging part is new). Such long GC pauses for the Par New
generation are strange, please check if you don't end up swapping the
JVM
(and also turn on bootstrap.mlockall, check the installation guide).

Also, where are you running elasticsearch on? Is it AWS? If so, which OS
are
you using?

Last, a bit more details can be obtained by turning on GC logging
information. You can head to bin/elasticsearch.in.sh, and uncomment the
-XX:+PrintGCDetails and the path to the GC logging file. What would be
interesting there is to see the details associated with such long
collections (and see if its actually user time being spent doing them).

On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz
thibaut.britz@trendiction.com wrote:

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore. But
still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might cause
this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.

We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?

Below the GC log of our servers:
Thanks,
Thibaut

r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]->[8.6gb]/[15.8gb]}{[CMS
Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total
[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[15.8gb]}{[CMS
Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total
[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total
[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total
[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total
[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]->[5.7gb]/[15.8gb]}{[CMS
Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code Cache]
[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total
[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.


(Shay Banon) #6

Can you try and use a newer JVM, 1.6.0_24 is pretty old.

On Mon, Jul 16, 2012 at 9:38 AM, Thibaut Britz <
thibaut.britz@trendiction.com> wrote:

Thanks for the hint.

We were hit by the leap second bug and already restarted all our
servers directly afterwards. Unfortunately, it's not related to that.

Thanks,
Thibaut

On Mon, Jul 16, 2012 at 3:41 PM, Romain filirom1@gmail.com wrote:

It may be because of the leap second:

http://unbxd.com/blog/2012/07/java-and-ksoftirqd-100-cpu-due-to-leap-second/

Try this:

https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/7h99SwBbRwo

Romain

2012/7/16 Thibaut Britz thibaut.britz@trendiction.com

Hi Shay,

Swap is turned off and no swap partition exists, bootstrap.mlockall is
enabled. They are all dedicated servers running on ubuntu 11.04, java
"1.6.0_24"
We will activate the extended GC logging and I will post the results.

The problem seems to mostly occur when we restart the cluster. So I
suspect it to to be related to a relatively high indexation volume
when the cluster is newly started.

Thanks,
Thibaut

On Sat, Jul 14, 2012 at 1:28 PM, Shay Banon kimchy@gmail.com wrote:

You might simply had it before as well, except that in 0.19, this is
logged
now (the logging part is new). Such long GC pauses for the Par New
generation are strange, please check if you don't end up swapping the
JVM
(and also turn on bootstrap.mlockall, check the installation guide).

Also, where are you running elasticsearch on? Is it AWS? If so, which
OS

are
you using?

Last, a bit more details can be obtained by turning on GC logging
information. You can head to bin/elasticsearch.in.sh, and uncomment
the

-XX:+PrintGCDetails and the path to the GC logging file. What would be
interesting there is to see the details associated with such long
collections (and see if its actually user time being spent doing
them).

On Fri, Jul 13, 2012 at 3:05 PM, Thibaut Britz
thibaut.britz@trendiction.com wrote:

Hi,

we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.

As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore.
But

still all requests to taht node will hang for a few minutes

I saw similar posts here in the last 2 weeks. Any idea what might
cause

this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.

We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?

Below the GC log of our servers:
Thanks,
Thibaut

r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code

Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS

Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools
{[Code

Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache]

[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s],
total

[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache]

[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools
{[Code

Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS

Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s],
total

[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s],
total

[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s],
total

[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen]
[7gb]->[7gb]/[15.8gb]}{[CMS

Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s],
total

[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code
Cache]

[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS

Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools
{[Code

Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools
{[Code

Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code
Cache]

[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m],
total

[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.


(system) #7