I just updated my cluster from 2.4.1. to 5.1.1 and everything went fine except that I'm seeing a lot of WARNings in the log. Basically it says' that garbage collection took too much time:
[2016-12-16T14:45:31,699][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1166] overhead, spent [3.3s] collecting in the last [3.6s]
[2016-12-16T14:45:46,258][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1178] overhead, spent [2.8s] collecting in the last [3.4s]
[2016-12-16T14:45:51,446][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1180] overhead, spent [3.9s] collecting in the last [4.1s]
[2016-12-16T14:46:04,802][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1190] overhead, spent [3.2s] collecting in the last [3.8s]
[2016-12-16T14:46:11,278][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1191] overhead, spent [6.1s] collecting in the last [6.4s]
[2016-12-16T14:46:18,649][WARN ][o.e.m.j.JvmGcMonitorService] [xxx] [gc][1195] overhead, spent [3.4s] collecting in the last [4.3s]
My cluster has:
- 2 data nodes with 24GB RAM, 10GB Heap for ES.
- 1 master-only node for quorum.
I never had any heap or JVM-related problems or error messages with 2.x before. Can anybody give me some pointer on why this might be happening?
I also seem to be having problems with Kibana 5.1.1 - it's running and apparently has access to Elasticsearch, because I see my previously saved objects. But if I try to get even the smallest amount of logs (like the last 5 or 10 minutes) from one of the "logstash-$DATE" indices I get a timeout after 30 seconds. Every time.