Elasticsearch G1GC tuning Memory increases continuously

Elastic Stack Elasticsearch
1

Hi Guys,

We are using Elasticsearch 8.2.0 with single node cluster, have added min/max heap as 30GB on a server with 128GB memory. We have around 1TB of index, the memory usage keeps on increasing in few hours and reaches 100%. It does not come down until we restart the Index, if anyone have faced such issue, please let me know the solutions.

We are using G1GC on a Java 17 version, mostly all default config in jvm file (except heap size)

Thanks in Advance
K

2

Welcome to our community! :smiley: We aren't all guys though.

Are you referring to jvm heap, or system memory here?

3

Thanks for the reply @warkolm

I am referring to system memory, it just keeps on increasing with the usage of elastic. Basically we have more aggregations in our application which queries elasticsearch. Within couple of hours of load, it reaches to full usage of heap (30GB) and system memory 128GB but does not come down until we restart the elasticsearch.

Not sure if the issue of GC not clearing memory in time. We are using G1GC with MaxGCPauseMills=5000.

Log show similar lines after few hours of usage but memory does not free up,
[Index] timer thread slept for [53.9s/53968ms] on absolute clock which is above the warn threshold of [5000ms]
[Index] [gc][young][10301][15] duration [872ms], collections [1]/[1.6s], total [872ms]/[1.8s], memory [19.9gb]->[3.2gb]/[30gb], all_pools {[young] [16.7gb]->[16mb]/[0b]}{[old] [1.9gb]->[2.2gb]/[30gb]}{[survivor] [1.2gb]->[1008mb]/[0b]}

4

That's pretty standard, it's the OS doing what it does to cache files that are commonly accessed.

5

I see 25-50%, i.e, 7-15GB of heap usage when I run _cat/nodes?v=true&h=name,node*,heap* but I can see system memory using 100gb from which ES is using 94gig. When i checked in Windows resource monitor then from 94gb which ES is using 61 is Shared set and 33 is Working set with many page faults, do you think its an issue with Windows or ES itself ?

Any pointers please?

6

Again, this is the OS caching files that it commonly accesses to make future access more efficient.
It's something that is managed by the OS, not Elasticsearch, and it's not a problem.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.