Hi,
we just upgraded our production cluster from 0.17 to 0.18 to 0.19.
The new zone features are really great, but we observed that nodes
were constantly marked as invalid and drop out.
As it turns out, the GC sometimes needs more than 2 or 3 minutes to
process. We increased the timeout so nodes don't drop out anymore. But
still all requests to taht node will hang for a few minutes
I saw similar posts here in the last 2 weeks. Any idea what might cause this?
We activated mlock (it would be good if there would a message in the
log saying that it's active) and used the default GC settings, but it
didn't help.
We didn't have any GC collection issues in the old version. Any ideas
what might cause this? We did however optimised the indexing process
so it's much faster. Could this be related to many documents being
indexed at the same time?
Below the GC log of our servers:
Thanks,
Thibaut
r6n1.
[2012-07-13 12:00:11,642][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][5992][15678] duration [787ms], collections [1]/[1.1s],
total [787ms]/[7.8m], memory [5gb]->[5gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[49.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[13.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen] [5gb]->[5gb]/[15.8gb]}{[CMS
Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:26,991][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6004][15722] duration [2.1s], collections [1]/[2.4s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[12mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[14.8mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:29,288][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6005][15723] duration [2.1s], collections [1]/[2.2s],
total [2.1s]/[7.9m], memory [5.2gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[7.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:00:48,606][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6017][15740] duration [7.9s], collections [1]/[7.9s],
total [7.9s]/[8.1m], memory [5.4gb]->[5.2gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[126mb]->[364kb]/[133.1mb]}{[Par Survivor Space]
[7.2mb]->[10.7mb]/[16.6mb]}{[CMS Old Gen]
[5.2gb]->[5.2gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:12,728][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6020][15753] duration [21.9s], collections [1]/[22s],
total [21.9s]/[8.4m], memory [5.4gb]->[5.4gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[14.3mb]->[2.7mb]/[16.6mb]}{[CMS Old Gen]
[5.4gb]->[5.4gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:38,008][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6041][15820] duration [2.5s], collections [3]/[3.4s],
total [2.5s]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[68.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[4.8mb]->[10mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:01:45,613][INFO
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6048][15831] duration [702ms], collections [1]/[1.2s],
total [702ms]/[8.5m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[100.3mb]->[4.8mb]/[133.1mb]}{[Par Survivor Space]
[8.4mb]->[7.2mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:02:25,657][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6065][15884] duration [23.3s], collections [1]/[23.9s],
total [23.3s]/[9m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[77.7mb]->[4.1mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[6.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:43,299][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6066][15885] duration [2.2m], collections [1]/[2.2m],
total [2.2m]/[11.2m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[4.1mb]->[747.9kb]/[133.1mb]}{[Par Survivor Space]
[6.8mb]->[16.5mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
[2012-07-13 12:04:59,157][WARN
][monitor.jvm][elasticsearch[r6n1][scheduler][T#1]] [r6n1]
[gc][ParNew][6067][15888] duration [15.4s], collections [3]/[15.8s],
total [15.4s]/[11.5m], memory [5.9gb]->[6gb]/[15.9gb], all_pools
{[Code Cache] [18.8mb]->[18.8mb]/[48mb]}{[Par Eden Space]
[747.9kb]->[388.7kb]/[133.1mb]}{[Par Survivor Space]
[16.5mb]->[16.6mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [60mb]->[60mb]/[82mb]}
r6n2.
r6n3.
r6n4.
r6n5.
r6n6.
[2012-07-13 13:10:26,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10101][36484] duration [20.8s], collections [3]/[21.8s],
total [20.8s]/[14.1m], memory [8.7gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[70.7mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[8.7mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:35,548][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10104][36492] duration [5.8s], collections [2]/[6.4s],
total [5.8s]/[14.2m], memory [8.6gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[18.7kb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[5.2mb]/[16.6mb]}{[CMS Old Gen]
[8.6gb]->[8.6gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:10:41,899][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10107][36503] duration [4.1s], collections [4]/[4.3s],
total [4.1s]/[14.3m], memory [8.8gb]->[8.6gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[110.6mb]->[115.6kb]/[133.1mb]}{[Par Survivor Space]
[6.1mb]->[2mb]/[16.6mb]}{[CMS Old Gen] [8.6gb]->[8.6gb]/[15.8gb]}{[CMS
Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:14,189][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10131][36605] duration [8.3s], collections [3]/[8.5s],
total [8.3s]/[14.5m], memory [8.7gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[439.9kb]->[15.2kb]/[133.1mb]}{[Par Survivor Space]
[6.9mb]->[7.1mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:11:50,448][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10137][36617] duration [30.7s], collections [1]/[30.9s],
total [30.7s]/[15m], memory [8.8gb]->[8.7gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[57.1mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[2.8mb]->[2.4mb]/[16.6mb]}{[CMS Old Gen]
[8.7gb]->[8.7gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:12:05,445][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10142][36636] duration [10.1s], collections [3]/[10.9s],
total [10.1s]/[15.2m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[2.4mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[558.9kb]->[300.6kb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:13:25,340][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10145][36642] duration [1.2m], collections [2]/[1.2m],
total [1.2m]/[16.4m], memory [8.8gb]->[8.8gb]/[15.9gb], all_pools
{[Code Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[25.5mb]->[126.8kb]/[133.1mb]}{[Par Survivor Space]
[3.9mb]->[3.5mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
[2012-07-13 13:14:56,007][WARN
][monitor.jvm][elasticsearch[r6n6][scheduler][T#1]] [r6n6]
[gc][ParNew][10159][36710] duration [1.2m], collections [3]/[1.2m],
total [1.2m]/[17.8m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [10.8mb]->[10.8mb]/[48mb]}{[Par Eden Space]
[78.8mb]->[1.3mb]/[133.1mb]}{[Par Survivor Space]
[6.3mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [43.1mb]->[43.1mb]/[82mb]}
r6n7.
r6n8.
[2012-07-13 12:43:21,850][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8663][12190] duration [1.1m], collections [3]/[1.1m],
total [1.1m]/[4.6m], memory [3.9gb]->[3.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[33.6mb]->[2.9mb]/[133.1mb]}{[Par Survivor Space]
[11mb]->[9.4mb]/[16.6mb]}{[CMS Old Gen]
[3.8gb]->[3.8gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:43:35,199][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8673][12200] duration [3s], collections [1]/[4s], total
[3s]/[4.7m], memory [3.9gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[77.9mb]->[1.8mb]/[133.1mb]}{[Par Survivor Space]
[12.9mb]->[9mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:02,539][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8693][12218] duration [6.4s], collections [1]/[6.5s],
total [6.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[73.3mb]->[1mb]/[133.1mb]}{[Par Survivor Space]
[8.2mb]->[8.8mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
[2012-07-13 12:44:40,213][WARN
][monitor.jvm][elasticsearch[r6n8][scheduler][T#1]] [r6n8]
[gc][ParNew][8727][12229] duration [2.4s], collections [1]/[3s], total
[2.4s]/[4.8m], memory [4gb]->[3.9gb]/[15.9gb], all_pools {[Code Cache]
[15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[124.3mb]->[177.4kb]/[133.1mb]}{[Par Survivor Space]
[9.9mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[3.9gb]->[3.9gb]/[15.8gb]}{[CMS Perm Gen] [50mb]->[50mb]/[82mb]}
r6n9.
r6n10.
r6n11.
r6n12.
r6n13.
r6n14.
r6n15.
r6n16.
r6n17.
r6n18.
r6n19.
r6n20.
r7n1.
[2012-07-13 11:01:45,714][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2669][4263] duration [4.2s], collections [3]/[5.1s],
total [4.2s]/[1m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[65mb]->[291.8kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:01:50,902][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2670][4268] duration [4.2s], collections [5]/[5.1s],
total [4.2s]/[1m], memory [5.8gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[291.8kb]->[8.9mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.8mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:00,444][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2679][4296] duration [806ms], collections [1]/[1.1s],
total [806ms]/[1.1m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[119mb]->[74.9mb]/[133.1mb]}{[Par Survivor Space]
[915.6kb]->[1mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:26,492][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2701][4329] duration [3.7s], collections [5]/[4.6s],
total [3.7s]/[1.1m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[776.8kb]->[969.6kb]/[133.1mb]}{[Par Survivor Space]
[449.2kb]->[1.5mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:36,529][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2709][4351] duration [2.8s], collections [1]/[2.8s],
total [2.8s]/[1.2m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[125mb]->[2.3mb]/[133.1mb]}{[Par Survivor Space]
[2mb]->[2.2mb]/[16.6mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[15.8gb]}{[CMS
Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:02:41,545][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2710][4353] duration [4.3s], collections [2]/[5s], total
[4.3s]/[1.3m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[2.3mb]->[5mb]/[133.1mb]}{[Par Survivor Space]
[2.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 11:04:57,122][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][2837][4601] duration [4.4s], collections [4]/[4.9s],
total [4.4s]/[1.4m], memory [6.2gb]->[6.1gb]/[15.9gb], all_pools
{[Code Cache] [14.1mb]->[14.1mb]/[48mb]}{[Par Eden Space]
[78.5mb]->[616.8kb]/[133.1mb]}{[Par Survivor Space]
[2.7mb]->[925.9kb]/[16.6mb]}{[CMS Old Gen]
[6.1gb]->[6.1gb]/[15.8gb]}{[CMS Perm Gen] [58.1mb]->[58.1mb]/[82mb]}
[2012-07-13 12:21:35,881][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7427][12288] duration [1.8s], collections [1]/[2s], total
[1.8s]/[3.5m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[129.3mb]->[3.4mb]/[133.1mb]}{[Par Survivor Space]
[10.4mb]->[9.7mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:21:46,480][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7433][12312] duration [4.8s], collections [1]/[5.2s],
total [4.8s]/[3.6m], memory [6.9gb]->[6.8gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[39.7mb]->[10.5mb]/[133.1mb]}{[Par Survivor Space]
[4.1mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[6.8gb]->[6.8gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:11,706][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7455][12347] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[3.7m], memory [6.9gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[27.8mb]->[3.2mb]/[133.1mb]}{[Par Survivor Space]
[8.3mb]->[6.7mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:38,482][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7479][12375] duration [2.1s], collections [1]/[2.7s],
total [2.1s]/[3.7m], memory [7gb]->[6.9gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[55.2mb]->[295.7kb]/[133.1mb]}{[Par Survivor Space]
[9.8mb]->[15.3mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:22:59,114][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7494][12407] duration [3.8s], collections [1]/[4.2s],
total [3.8s]/[3.8m], memory [7.1gb]->[6.9gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[131.2mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[1.6mb]->[2.6mb]/[16.6mb]}{[CMS Old Gen]
[6.9gb]->[6.9gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:11,637][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7505][12421] duration [1s], collections [1]/[1.1s], total
[1s]/[3.9m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[63.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1mb]->[987.7kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:44,223][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7534][12453] duration [1.7s], collections [1]/[2s], total
[1.7s]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[16.8kb]/[133.1mb]}{[Par Survivor Space]
[1.4mb]->[952.2kb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:23:49,542][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7539][12462] duration [724ms], collections [1]/[1.2s],
total [724ms]/[4m], memory [7gb]->[7gb]/[15.9gb], all_pools {[Code
Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[2.5mb]->[2.7mb]/[133.1mb]}{[Par Survivor Space]
[913.9kb]->[1mb]/[16.6mb]}{[CMS Old Gen] [7gb]->[7gb]/[15.8gb]}{[CMS
Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:48,348][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7584][12531] duration [2.2s], collections [2]/[2.9s],
total [2.2s]/[4.3m], memory [7.1gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[1.2mb]->[1.6mb]/[133.1mb]}{[Par Survivor Space]
[736.2kb]->[783.1kb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:52,591][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7588][12536] duration [1s], collections [1]/[1.1s], total
[1s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools {[Code Cache]
[15mb]->[15mb]/[48mb]}{[Par Eden Space]
[92.1mb]->[67.8kb]/[133.1mb]}{[Par Survivor Space]
[899.7kb]->[2mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:24:58,726][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7591][12543] duration [3.2s], collections [2]/[3.8s],
total [3.2s]/[4.3m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120.9mb]->[9mb]/[133.1mb]}{[Par Survivor Space]
[5.1mb]->[4.3mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:19,635][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7605][12580] duration [6.8s], collections [1]/[6.9s],
total [6.8s]/[4.5m], memory [7.2gb]->[7.1gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[105.4mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.9mb]/[16.6mb]}{[CMS Old Gen]
[7.1gb]->[7.1gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:46,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7624][12607] duration [5.2s], collections [1]/[5.3s],
total [5.2s]/[4.6m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[115.8mb]->[12.5kb]/[133.1mb]}{[Par Survivor Space]
[862kb]->[548.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 12:25:53,811][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][7627][12614] duration [4.7s], collections [1]/[4.7s],
total [4.7s]/[4.7m], memory [1.8gb]->[1.7gb]/[15.9gb], all_pools
{[Code Cache] [15mb]->[15mb]/[48mb]}{[Par Eden Space]
[120mb]->[2.2mb]/[133.1mb]}{[Par Survivor Space]
[689.2kb]->[680.3kb]/[16.6mb]}{[CMS Old Gen]
[1.7gb]->[1.7gb]/[15.8gb]}{[CMS Perm Gen] [58.3mb]->[58.3mb]/[82mb]}
[2012-07-13 13:07:55,134][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10140][16454] duration [5.6s], collections [5]/[6.4s],
total [5.6s]/[5.6m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[99.4mb]->[357.7kb]/[133.1mb]}{[Par Survivor Space]
[6.7mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:06,093][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10149][16472] duration [2.3s], collections [1]/[2.7s],
total [2.3s]/[5.6m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[34.4mb]->[572.1kb]/[133.1mb]}{[Par Survivor Space]
[1.3mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:31,880][INFO
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10171][16514] duration [3.6s], collections [4]/[4.4s],
total [3.6s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[26.5mb]->[483.4kb]/[133.1mb]}{[Par Survivor Space]
[3mb]->[797.2kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:35,299][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10172][16516] duration [3.2s], collections [2]/[3.4s],
total [3.2s]/[5.7m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[483.4kb]->[551.4kb]/[133.1mb]}{[Par Survivor Space]
[797.2kb]->[5mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:41,541][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10176][16527] duration [2.4s], collections [2]/[3.2s],
total [2.4s]/[5.8m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[6mb]->[28.8kb]/[133.1mb]}{[Par Survivor Space]
[300.9kb]->[374.1kb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:48,737][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10180][16535] duration [3.3s], collections [2]/[4.1s],
total [3.3s]/[5.8m], memory [5.7gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[68.2mb]->[50.6kb]/[133.1mb]}{[Par Survivor Space]
[1.7mb]->[1.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:08:55,241][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10185][16551] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[5.9m], memory [5.6gb]->[5.6gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[52.9kb]->[16.5kb]/[133.1mb]}{[Par Survivor Space]
[2.4mb]->[3.2mb]/[16.6mb]}{[CMS Old Gen]
[5.6gb]->[5.6gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:24,266][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10212][16597] duration [1.9s], collections [1]/[2.2s],
total [1.9s]/[5.9m], memory [5.7gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[206.9kb]/[133.1mb]}{[Par Survivor Space]
[3.2mb]->[2.3mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:09:59,338][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10238][16632] duration [9.1s], collections [5]/[9.9s],
total [9.1s]/[6.1m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[97.6mb]->[1.9mb]/[133.1mb]}{[Par Survivor Space]
[2.1mb]->[1.6mb]/[16.6mb]}{[CMS Old Gen]
[5.7gb]->[5.7gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:07,579][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10243][16645] duration [3.3s], collections [2]/[4.2s],
total [3.3s]/[6.2m], memory [5.8gb]->[5.7gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[58.9mb]->[5.8mb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1mb]/[16.6mb]}{[CMS Old Gen] [5.7gb]->[5.7gb]/[15.8gb]}{[CMS
Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:30,327][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10259][16670] duration [6.6s], collections [1]/[7.1s],
total [6.6s]/[6.3m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[44.2mb]->[2.1mb]/[133.1mb]}{[Par Survivor Space]
[633.6kb]->[685.6kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:10:40,008][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10266][16688] duration [1.9s], collections [1]/[2.3s],
total [1.9s]/[6.4m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[42.9mb]->[174.3kb]/[133.1mb]}{[Par Survivor Space]
[1.1mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:22,592][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10306][16739] duration [1.5s], collections [1]/[1.7s],
total [1.5s]/[6.4m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[50.5mb]->[3.3mb]/[133.1mb]}{[Par Survivor Space]
[2.9mb]->[1.1mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:35,865][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10309][16746] duration [10.8s], collections [1]/[11.1s],
total [10.8s]/[6.6m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[37.6mb]->[7.5mb]/[133.1mb]}{[Par Survivor Space]
[181.5kb]->[133.3kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:37,723][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10310][16747] duration [1.3s], collections [1]/[1.8s],
total [1.3s]/[6.6m], memory [5.8gb]->[5.8gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[7.5mb]->[18.3kb]/[133.1mb]}{[Par Survivor Space]
[133.3kb]->[540kb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:47,016][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10316][16758] duration [3s], collections [2]/[3.8s],
total [3s]/[6.7m], memory [5.9gb]->[5.8gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[63.3mb]->[3.6mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.3mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.8gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:11:55,906][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10323][16771] duration [2.3s], collections [2]/[2.6s],
total [2.3s]/[6.7m], memory [6gb]->[5.9gb]/[15.9gb], all_pools {[Code
Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[126.9mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[1.5mb]->[1.4mb]/[16.6mb]}{[CMS Old Gen]
[5.8gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
[2012-07-13 13:12:23,075][WARN
][monitor.jvm][elasticsearch[r7n1][scheduler][T#1]] [r7n1]
[gc][ParNew][10348][16806] duration [1.1s], collections [1]/[1.4s],
total [1.1s]/[6.8m], memory [5.9gb]->[5.9gb]/[15.9gb], all_pools
{[Code Cache] [15.6mb]->[15.6mb]/[48mb]}{[Par Eden Space]
[1.1mb]->[1.2mb]/[133.1mb]}{[Par Survivor Space]
[184.2kb]->[237.9kb]/[16.6mb]}{[CMS Old Gen]
[5.9gb]->[5.9gb]/[15.8gb]}{[CMS Perm Gen] [58.4mb]->[58.4mb]/[82mb]}
r7n2.
r7n3.
[2012-07-13 10:44:24,127][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1628][3214] duration [10.2s], collections [2]/[10.3s],
total [10.2s]/[1m], memory [8.9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[108.3mb]->[24mb]/[133.1mb]}{[Par Survivor Space]
[10.9mb]->[7.3mb]/[16.6mb]}{[CMS Old Gen]
[8.8gb]->[8.8gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:02,249][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1630][3225] duration [36.5s], collections [4]/[37s],
total [36.5s]/[1.6m], memory [9gb]->[8.9gb]/[15.9gb], all_pools {[Code
Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95mb]->[9.3mb]/[133.1mb]}{[Par Survivor Space]
[11.9mb]->[5.8mb]/[16.6mb]}{[CMS Old Gen]
[8.9gb]->[8.9gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:45:43,819][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1657][3339] duration [15.1s], collections [1]/[15.2s],
total [15.1s]/[1.9m], memory [9.4gb]->[9.3gb]/[15.9gb], all_pools
{[Code Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[121.2mb]->[13.1mb]/[133.1mb]}{[Par Survivor Space]
[7.9mb]->[15.5mb]/[16.6mb]}{[CMS Old Gen]
[9.3gb]->[9.3gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:46:01,557][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1665][3371] duration [10.6s], collections [1]/[10.6s],
total [10.6s]/[2.1m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools
{[Code Cache] [13mb]->[13mb]/[48mb]}{[Par Eden Space]
[124.9mb]->[309.7kb]/[133.1mb]}{[Par Survivor Space]
[7.1mb]->[7.9mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.8mb]->[50.8mb]/[82mb]}
[2012-07-13 10:49:05,877][WARN
][monitor.jvm][elasticsearch[r7n3][scheduler][T#1]] [r7n3]
[gc][ParNew][1668][3387] duration [3m], collections [6]/[3m], total
[3m]/[5.2m], memory [9.5gb]->[9.4gb]/[15.9gb], all_pools {[Code Cache]
[13mb]->[13mb]/[48mb]}{[Par Eden Space]
[95.9mb]->[1.1mb]/[133.1mb]}{[Par Survivor Space]
[15.7mb]->[12mb]/[16.6mb]}{[CMS Old Gen]
[9.4gb]->[9.4gb]/[15.8gb]}{[CMS Perm Gen] [50.9mb]->[50.9mb]/[82mb]}
r7n4.
r7n5.
r7n6.
r7n7.
r7n8.
r7n9.
r7n10.
r7n11.
r7n12.
r7n13.
r7n14.
r7n15.
r7n16.
r7n17.
r7n18.
[2012-07-13 10:32:32,003][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][922][686] duration [4.1s], collections [1]/[4.2s], total
[4.1s]/[9.2s], memory [1.4gb]->[1.3gb]/[15.9gb], all_pools {[Code
Cache] [10.5mb]->[10.5mb]/[48mb]}{[Par Eden Space]
[59.6mb]->[3.9mb]/[133.1mb]}{[Par Survivor Space]
[10.1mb]->[9.5mb]/[16.6mb]}{[CMS Old Gen]
[1.3gb]->[1.3gb]/[15.8gb]}{[CMS Perm Gen] [56.3mb]->[56.3mb]/[82mb]}
[2012-07-13 10:37:14,787][WARN
][monitor.jvm][elasticsearch[r7n18][scheduler][T#1]] [r7n18]
[gc][ParNew][1026][751] duration [2.7m], collections [1]/[2.7m], total
[2.7m]/[2.9m], memory [1.5gb]->[1.5gb]/[15.9gb], all_pools {[Code
Cache] [10.6mb]->[10.6mb]/[48mb]}{[Par Eden Space]
[1.3mb]->[2mb]/[133.1mb]}{[Par Survivor Space]
[6.6mb]->[13.3mb]/[16.6mb]}{[CMS Old Gen]
[1.5gb]->[1.5gb]/[15.8gb]}{[CMS Perm Gen] [56.7mb]->[56.7mb]/[82mb]}
r7n19.
r7n20.