Elastic Stack out of Memory

Elastic Stack Elasticsearch
1

I am having an issue running some kibana reports. It runs out of memory (See below), I started with basic memoryu, upped it to 8GB and now to 16GB. What may be going on?

[2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_logstash_version_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_kibana_version_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_elasticsearch_nodes], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_xpack_license_expiration], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_elasticsearch_cluster_status], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_elasticsearch_version_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:02:46,072][ERROR][o.e.x.m.c.n.NodeStatsCollector] [node-1] collector [node_stats] timed out when collecting data
[2018-07-25T10:02:46,076][WARN ][o.e.x.w.e.ExecutionService] [node-1] failed to execute watch [J8OVNfqLSe2NW0TOJPOhzw_logstash_version_mismatch]
[2018-07-25T10:02:46,086][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][1050][14] duration [26.5s], collections [1]/[27.1s], total [26.5s]/[4.8m], memory [15.8gb]->[15.8gb]/[15.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [51.3mb]->[57.2mb]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2018-07-25T10:02:46,086][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][1050] overhead, spent [26.5s] collecting in the last [27.1s]
[2018-07-25T10:03:09,939][ERROR][o.e.x.m.c.i.IndexStatsCollector] [node-1] collector [index-stats] timed out when collecting data
[2018-07-25T10:03:09,946][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][1051][15] duration [23.5s], collections [1]/[23.8s], total [23.5s]/[5.2m], memory [15.8gb]->[15.8gb]/[15.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [57.2mb]->[74.2mb]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2018-07-25T10:03:09,946][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][1051] overhead, spent [23.5s] collecting in the last [23.8s]
[2018-07-25T10:03:36,701][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_kibana_version_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:03:36,701][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_logstash_version_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:03:36,701][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_xpack_license_expiration], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:03:36,702][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_elasticsearch_nodes], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:03:36,702][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_elasticsearch_version_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:03:36,703][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [node-1] collector [cluster_stats] timed out when collecting data
[2018-07-25T10:03:36,702][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw_elasticsearch_cluster_status], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.]
[2018-07-25T10:03:36,704][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][1052][16] duration [26.5s], collections [1]/[26.7s], total [26.5s]/[5.6m], memory [15.8gb]->[15.8gb]/[15.8gb], all_pools {[young] [865.3mb]->[865.3mb]/[865.3mb]}{[survivor] [74.2mb]->[84.8mb]/[108.1mb]}{[old] [14.9gb]->[14.9gb]/[14.9gb]}
[2018-07-25T10:03:36,705][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][1052] overhead, spent [26.5s] collecting in the last [26.7s]

2

What about some of the other problems: logstash_version_mismatch, kibana_version_mismatch, xpack_license_expiration, elasticsearch_version_mismatch.

3

I dont know where they are coming from or if they matter in this case.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.