GC Collector Error shut down ingest node


(Emiliano Baum) #1

The elastic service shut down when the log report the following error.
I use ES 5.6, over Ubuntu server 16.04. The cluster have 10 nodes, two of this are ingestion nodes, and one of this have the problem.
this is the service log:

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: signal) since Tue 2018-09-18 22:00:33 UTC; 13h ago

and this is the log of the application:

[2018-09-18T21:59:51,874][INFO ][o.e.m.j.JvmGcMonitorService] [MON-CGSSESCluster01IngestNode2-S] [gc][117706] overhead, spent [313ms] collecting in the last [1s]
[2018-09-18T21:59:56,685][INFO ][o.e.m.j.JvmGcMonitorService] [MON-CGSSESCluster01IngestNode2-S] [gc][young][117710][20382] duration [917ms], collections [1]/[1.7s], total [917ms]/[23.6m], memory [5.7gb]->[5.7gb]/[9.9gb], all_pools {[young] [288.4mb]->[43.3mb]/[532.5mb]}{[survivor] [66.5mb]->[66.5mb]/[66.5mb]}{[old] [5.4gb]->[5.5gb]/[9.3gb]}
[2018-09-18T21:59:56,688][WARN ][o.e.m.j.JvmGcMonitorService] [MON-CGSSESCluster01IngestNode2-S] [gc][117710] overhead, spent [917ms] collecting in the last [1.7s]
[2018-09-18T22:00:00,836][INFO ][o.e.m.j.JvmGcMonitorService] [MON-CGSSESCluster01IngestNode2-S] [gc][117714] overhead, spent [443ms] collecting in the last [1.1s]

(Mark Walkom) #2

Can you post most of the log, there's nothing there that suggests that the node was shutdown.


(Emiliano Baum) #3

Hi warkholm, the elastic log don't say to much, only the lines i pasted before. The service status said the proccess was killed at the same time the elastic log report the overhead warning.
This error could be caused for a lot of querys to the node?

I have two dedicated ingestion nodes that mainly receive indexing messages. To monitor the cluster, I use "Cerebro", and I suspect that the problem is due to this software, because yesterday I lowered the refresh rate and until now the cluster does not report problems


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.