Elasticsearch Master Node JVM reaching 99%

Elastic Stack Elasticsearch
1

I am running a 3 Master 8 Data node cluster.
Elasticsearch Version 7.8.0

The cluster bumps along happily for several days with the regular sawtooth JVM pattern then suddenly ramps up and then hits 99%.

image
image725×215 14.7 KB

This leads to issues performing normal operations. A restart puts the memory back to normal levels but I can't figure out what causes it. I have seen before that a memory leak in a previous version caused similar issues but I would have thought that the GC would keep the JVM usage within comfortable limits.

2

Would you capture a heap dump the next time the heap maxes out like that? Let us know when you've got it and we'll arrange a way to send it in.

3

@DavidTurner It has happened again on all 3 of my master nodes. I see there is now a 7.8.1, might that fix it?

4

I doubt it since I don't think anything has changed recently in this area. I asked for heap dumps above -- without them, we can't really say what the problem is, and without knowing that we can't really say anything about a fix.

5

I have captured one. What is the best way to send it to you.

6

Great, thanks. I've sent you a direct message with an upload link.

7

image
image1626×352 113 KB

Looks to me like you're using the third-party ReadonlyREST plugin and it's gone haywire, but I can't say much more than that as it's not something we support. I recommend using the official security features instead, they're included by default, they don't cost anything and AFAIK don't suffer from this kind of problem.

1 Like
8

Thanks for looking into it. I am keeping an eye on the native security. At the moment, we are making use of IP filtering and LDAP integration, which we would lose if we went to stack native security. Although, if this lasts too long, a loss of convenience outweighs a stable cluster.

Thanks,

Richard

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.