Out of memory exception for coordinating nodes

Elastic Stack Elasticsearch
1

I've a ES cluster 6.6.2 composed of 3 master, 4 datanodes, 2 coordinating nodes.
2 kibana instances connecting to 2 coordinating nodes.

2 coordinating nodes are 16GB of RAM, and 7GB of Heap (min=max heap).
On coordinating nodes only I face "out of memory exception" and O.S kill jvm process for elastichsearch.

I've enabled the bootstrap memory lock, and followed documentations.

https://www.elastic.co/guide/en/elasticsearch/reference/6.6/_memory_lock_check.html

Any idea what to do to prevent out-of-memory and keep service is up?

2

Two questions:

First: What kind of queries are you sending to those nodes? Are your querying a lot of shards in a single query or paginating deep?

Second: How exactly is the Elasticsearch process being killed? Can you share log file output, and is the kernel OOM killer doing this or is there an exception in the logs?

3

Yes, the queries are hitting several shards, usually kibana is hitting different indexes.
Most of works on Kibana.

from systemd:
Jan 21 15:08:40 elk.novalocal systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
Jan 21 15:08:40 elk.novalocal systemd[1]: Unit elasticsearch.service entered failed state.
Jan 21 15:08:40 elk.novalocal systemd[1]: elasticsearch.service failed.

Status for monitoring.

image
image1132×290 76.2 KB

4

I see the heap is never reached to the maximum? so why the service stops itself.

5

how is the service stopped? Please check the ES logfile and the dmesg output... are there any mentions?

--Alex

6

From dmesg

Jan 21 15:08:40 elk.novalocal systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
Jan 21 15:08:40 elk.novalocal systemd[1]: Unit elasticsearch.service entered failed state.
Jan 21 15:08:40 elk.novalocal systemd[1]: elasticsearch.service failed.
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.