High memory usage

Elastic Stack Elasticsearch
1

Hello,

I'm new to the ElasticStacks universe and would like to ask a question.

Below is a screenshot of the resources being used by my nodes.

image
image626×167 47.8 KB

You can see that memory usage is high. Is this normal?

Should I be concerned? Some people say not to worry, and others have told me that this can cause a circuit_breaker exception.

Thanks!

2

High RAM usage generally just mean that the operating system is using the memory to cache files, which is important for optimal performance. This is nothing to be concerned about and completely normal.

This can happen if heap usage is high, not RAM.

3

Hi @Christian_Dahlqvist thanks for your attention!

In my case, my cluster started triggering numerous circuit breaker exception alerts, example below:

image
image627×576 116 KB

And this raises a lot of questions for me. In these cases, should I increase my memory?

Or should I take other action?

In the JVM pressure graphs, the peak is always 80%, I think it's ok... but why then am I having so many cases of circuit_breaker?

Thanks so much!

4

It looks like you are hitting heap limits. The first error message indicates that you are performing updates. Can you provide some additional details about this?

Which version of Elasticsearch are you using?

What is the average size of the documents you are updating?

Are you using nested documents or parent-child relationships?

Are you updating using bulk requests? If so, how large are the bulk requests and how many concurrent requests are you sending?

What is the size and specification of the nodes in the cluster?

5

Which version of Elasticsearch are you using?

I'm using open-search, I'm really motivated to move away from open-search and create an independent cluster, do you recommend it?

What is the average size of the documents you are updating?

It's hard to say because I have millions of documents in the cluster, in this case where the circuit_breaker occurred the document size was 3.57 KB. I have some documents bigger than this, but i dont think the average are "too big".

Are you using nested documents or parent-child relationships?

Yes, nested I always use, parent-child never.

In this case where the circuit_breaker error occurred, the document contained neither nested nor parent-child.

Are you updating using bulk requests? If so, how large are the bulk requests and how many concurrent requests are you sending?

Good question, yes in my system we have several bulk requests, I also wouldn't know how big these requests are and how many are occurring at the same time. I will try to collect this information.

What is the size and specification of the nodes in the cluster?

image
image1090×133 35.2 KB

I obtained more information from the cluster, as shown in the image above.

Despite my limited experience with Elastic, I truly believe that the reason the cluster are triggering so many circuit_breaker requests has more to do with how I'm using Elastic than with my cluster's configuration, which I think is "normal/good."

I think the biggest problem is that Elastic is the single source of truth for my entire system. I don't think Elastic is designed for that. This means I have thousands of read/write operations occurring simultaneously. Do you agree with this?

I really appreciate your attention.

6

Opensearch and Elasticsearch share heritage as Opensearch started as a fork. Nowadays they have however grown apart quite sigificantly and are basically different products. I would recommend that you contact the Opensearch community as they will likely better be able to assist.

1 Like