ES 5.1.1 spending too much time in GC

Elastic Stack Elasticsearch
1

I just updated my cluster from 2.4.1. to 5.1.1 and everything went fine except that I'm seeing a lot of WARNings in the log. Basically it says' that garbage collection took too much time:

[2016-12-16T14:45:31,699][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1166] overhead, spent [3.3s] collecting in the last [3.6s]
[2016-12-16T14:45:46,258][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1178] overhead, spent [2.8s] collecting in the last [3.4s]
[2016-12-16T14:45:51,446][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1180] overhead, spent [3.9s] collecting in the last [4.1s]
[2016-12-16T14:46:04,802][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1190] overhead, spent [3.2s] collecting in the last [3.8s]
[2016-12-16T14:46:11,278][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1191] overhead, spent [6.1s] collecting in the last [6.4s]
[2016-12-16T14:46:18,649][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1195] overhead, spent [3.4s] collecting in the last [4.3s]

My cluster has:

  • 2 data nodes with 24GB RAM, 10GB Heap for ES.
  • 1 master-only node for quorum.

I never had any heap or JVM-related problems or error messages with 2.x before. Can anybody give me some pointer on why this might be happening?

I also seem to be having problems with Kibana 5.1.1 - it's running and apparently has access to Elasticsearch, because I see my previously saved objects. But if I try to get even the smallest amount of logs (like the last 5 or 10 minutes) from one of the "logstash-$DATE" indices I get a timeout after 30 seconds. Every time.

2

Okay, scratch that.

I didn't notice, that the JVM parameters moved from /etc/default/elasticsearch to /etc/elasticsearch/jvm.options and my 10G ES_HEAP setting in the former was simply ignored.

I set the correct values in /etc/elasticsearch/jvm.options, now ES is starting with 10G heap and no GC problems.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.