High Heap utilization in Client Node

Elastic Stack Elasticsearch
1

We are seeing high heap utilization in two of the client nodes. Its configured as 4GB RAM and its hitting around 98%. There is nothing showing in elasticsearch logs.
ES LOG:

[2017-12-12T19:23:48,005][WARN ][o.e.m.j.JvmGcMonitorService] [zlt10363.vci.att.com-01] [gc][old][6778036][135594] duration [17.8s], collections [1]/[18.3s], total [17.8s]/[5.4h], memory [3.9gb]->[3.6gb]/[3.9gb], all_pools {[young] [266.2mb]->[14.4mb]/[266.2mb]}{[survivor] [32.6mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2017-12-12T19:23:48,005][WARN ][o.e.m.j.JvmGcMonitorService] [zlt10363.vci.att.com-01] [gc][6778036] overhead, spent [17.8s] collecting in the last [18.3s]

But I can see something related to it in Kibana logs.

{"type":"log","@timestamp":"2017-12-12T23:25:59Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-12T23:25:59Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-12T23:28:53Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-12T23:28:53Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-12T23:30:59Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-12T23:30:59Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-12T23:39:59Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-12T23:39:59Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-12T23:42:58Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-12T23:42:58Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-12T23:57:47Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-12T23:57:47Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-13T00:03:30Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-13T00:03:30Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2017-12-13T00:06:22Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
{"type":"log","@timestamp":"2017-12-13T00:06:22Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}

Could you please help me out in fixing this issue. Thanks.

2

This may be of some help https://www.linuxatemyram.com/ :slight_smile:

3

Thanks for sharing the link. But this is happening from last 2 days. Previously, it used to work fine. But now, its reaching 99% and I can see that elasticsearch plugin is red in kibana logs. I think its not normal as Elasticsearch plugin is becoming red often.

     {"type":"log","@timestamp":"2017-12-13T10:55:46Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T11:12:10Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T11:12:10Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T11:32:50Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T11:32:50Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T11:57:35Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T11:57:35Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T12:00:16Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T12:00:16Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T12:34:13Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T12:34:13Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T14:12:44Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T14:12:44Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T14:52:07Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T14:52:07Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T14:54:56Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T14:54:56Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T15:00:28Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T15:00:28Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T15:02:30Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T15:02:30Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T15:03:02Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T15:03:02Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
    {"type":"log","@timestamp":"2017-12-13T15:05:10Z","tags":["status","plugin:elasticsearch@5.2.1","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Kibana index ready"}
    {"type":"log","@timestamp":"2017-12-13T15:05:10Z","tags":["status","ui settings","error"],"pid":20308,"state":"red","message":"Status changed from green to red - Elasticsearch plugin is red","prevState":"green","prevMsg":"Ready"}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.