Robin_Guo
December 11, 2017, 11:19am
1
Hi Folks,
we have 300 servers send metrics to logstash ,and then go to elasticsearch.
PS.
171211191416 root@esdata02 ~ # jstat -gcutil 28948 2000 100
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 100.00 80.26 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 80.35 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 83.33 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 83.71 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 83.99 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 84.17 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 84.54 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 84.82 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 85.20 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 85.47 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 85.47 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 85.57 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 85.85 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 86.22 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 86.31 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 86.69 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 86.78 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 87.15 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 87.43 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 87.62 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 87.90 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 88.27 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 88.36 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 88.55 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 88.64 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 88.73 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 89.11 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 89.20 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 89.57 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 89.76 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 89.94 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 90.04 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 93.39 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 93.67 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 94.04 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
0.00 100.00 94.23 81.35 91.53 82.89 2056 219.945 0 0.000 219.945
colings86
December 11, 2017, 11:41am
2
I don't see any problem in this output. No GC has happened during this output (since YGC and FGC have not increased) so I would not expect S1 to change. The eden space (E) is increasing and close to being full so I would expect a young GC to happen soon after the end of this output at which point I would expect S1 to drop to 0 (as any surviving object will be moved to S0) and S0 to increase to some value between 0 and 100 depending on the survival rate of the objects in the Eden space.
Also note that G1GC is currently not supported in Elasticsearch though we are continually testing G1GC to determine if/when we are able to support it.
Robin_Guo
December 12, 2017, 2:54am
3
hi @colings86 ,
I've been monitoring the GC for a while, even if the YGC changed , but S1 never changed , it always 100, I never ever see the s1 changed to some valule.
171212105051 root@esdata02 ~ # jstat -gcutil 28948 2000 100
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 100.00 75.41 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 76.08 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 76.17 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 76.27 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 76.56 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 76.65 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 77.03 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 77.22 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 77.32 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 77.89 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 78.09 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 78.18 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 78.47 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 78.56 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 78.85 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 79.04 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 79.23 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 79.52 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 79.71 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 79.90 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 80.00 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 80.00 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 80.29 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 80.67 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 82.58 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 84.02 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 84.50 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 84.78 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 85.17 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 87.94 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 88.42 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 89.00 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 89.28 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 93.30 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 94.93 82.10 91.54 82.89 2191 229.038 0 0.000 229.038
0.00 100.00 8.53 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
0.00 100.00 12.40 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
0.00 100.00 13.95 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
0.00 100.00 15.50 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
0.00 100.00 21.71 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
0.00 100.00 28.68 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
0.00 100.00 29.46 55.35 91.54 82.89 2192 229.149 0 0.000 229.149
I think this is a visualization problem of jstat. The G1 heap layout is quite different to heap layouts of previous garbage collectors and the jstat tool seems not to have been adapted.
If you want to analyze GC behavior the better option is to enable GC logs and to look at them instead. If you are interested in detailed information about region sizes you can turn add -XX:+PrintHeapAtGC to your JVM options.
For a more thorough explanation of how G1 works, you can watch e.g. the following video by Monica Beckwith:
system
January 9, 2018, 10:30am
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.