GC spending too much time

Elastic Stack Elasticsearch
1

Hello everyone!

I am running Elasticsearch at version 5.1.1.
I observed that him is spending too much time with garbage collector task, as you can see bellow:

[2017-01-17T11:18:50,156][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51104] overhead, spent [293ms] collecting in the last [1s]
[2017-01-17T11:18:57,158][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51111] overhead, spent [264ms] collecting in the last [1s]
[2017-01-17T11:19:15,497][WARN ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51115] overhead, spent [14.6s] collecting in the last [15.3s]
[2017-01-17T11:19:25,715][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51125] overhead, spent [356ms] collecting in the last [1s]
[2017-01-17T11:19:45,095][WARN ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51130] overhead, spent [14.7s] collecting in the last [15.2s]
[2017-01-17T11:20:07,698][WARN ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51139] overhead, spent [14.1s] collecting in the last [14.3s]
[2017-01-17T11:20:17,949][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51149] overhead, spent [393ms] collecting in the last [1s]
[2017-01-17T11:20:24,977][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51156] overhead, spent [302ms] collecting in the last [1s]
[2017-01-17T11:20:40,270][WARN ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51157] overhead, spent [15s] collecting in the last [15.2s]
[2017-01-17T11:20:50,555][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51167] overhead, spent [426ms] collecting in the last [1s]
[2017-01-17T11:21:10,140][WARN ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51172] overhead, spent [15.3s] collecting in the last [15.3s]
[2017-01-17T11:21:20,434][INFO ][o.e.m.j.JvmGcMonitorService] [SCCHIB4ESCB-07] [gc][51182] overhead, spent [311ms] collecting in the last [1s]

I've defined correctly my jvm.options file with 8g for heap memory space.

The logs above are from my master node (currently he is running as data node too yet. I'll fix it soon).

What can i do to improve the GC performance?

2

It's hard to say without more information about what is consuming your heap and what you're doing. Can you provide a heap dump (be careful that it might contain sensitive data; if you need to share it privately we can arrange that)?

3

I tryed to make the heap dump, but it's taking a long time. I'll try again soon.

Anyway, there's another metric can i take to help the analysis?

Thank you!

4

Even a heap histogram would help here.

5

Hi @jasontedor.

I think my environment is too busy.

I tryed to generate the heap histogram, but got an exception.

[root@scchib4escb-07 ~]# jmap -F -histo 31436
Attaching to process ID 31436, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.tools.jmap.JMap.runTool(JMap.java:197)
        at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 24.111-b01. Target VM is 25.111-b15
        at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:234)
        at sun.jvm.hotspot.runtime.VM.<init>(VM.java:297)
        at sun.jvm.hotspot.runtime.VM.initialize(VM.java:368)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:598)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
        at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
        at sun.jvm.hotspot.tools.ObjectHistogram.main(ObjectHistogram.java:56)
6

It looks like you're using JDK 7 targeting JDK 8. Sometimes this work, sometimes it doesn't. You should just use the same JDK that the VM is running on.

7

Its not the exact problem. But in my case it happend. Are you using any from & to in your query upon a big index??? If yes take out those quiries and check.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.