What causes high node Heap Memory more than 75%?

Elastic Stack Elasticsearch
1

Hi Team,

I am receiving high memory alerts on our cluster, we are hosted on "cloud.elastic.co".

3 Nodes - 2 (data/master eligible node) + 1 tie breaker node.

Total 28 GB of data on a single index which has 5 shards and 1 replica set.

Along with causes could you also help me finding out solution of what should be our next steps in such a situation?

Thanks,

2

What is the output of:

GET /_cat/health?v
GET /_cat/indices?v
GET /_cat/shards?v
3

What is the size of the cluster? Are you using nested documents and/or parent-child relationships?

4

@Christian_Dahlqvist We are using nested documents heavily as we have such use case. Cluster size AFAIK it is 8GB RAM 2 Data node and 1 tiebreaker node.

@dadoonet I have limited access to the Kibana but I am surely going to share requested details. But in a meanwhile can you brief me what sort of those parameters will help to isolate the issue?

5

Hello @dadoonet,

Health is showing normal in green state with 7 shards one 3 indexes.

Indices - 1 for kibana, 1 for security and 1 is our production instance, total 3.

Shards - All shards evenly distributed on both the nodes but I see few shards are heavily used while other are not much in use.

Let me know your inputs based on above parameters.

Hey @Christian_Dahlqvist,

Did you get chance to look into above situation? Let me know if you have any inputs.

Thanks,

6

Nested documents use more heap that flat documents, but I do not have any figures. Will have to leave that for someone with more experience of that to comment.

7

I'd like to see the outputs of the commands please.
Did you try to get support from https://www.elastic.co/cloud/as-a-service/support ?

8

Thanks @Christian_Dahlqvist appreciate your help.

9

Hi @dadoonet,

We talked to our support team, but as it is basic support we are not getting actual reasons from team.

Regarding output of commands I am checking with my DevOps team as it is production sensitive so not sure about sharing it.

What parameters will you think of getting more meory?

  • Segment merging
  • Large no. of nested documents
  • GC run

I wanted to know which parameters causes this kind of issues, from monitoring chart I am not able to co-relate the patterns of high CPU usage.

Let me know if you have any suggetions.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.