Ubuntu 20.04 ELK 8.10.2 Consuming all the RAM

[RobertC1 (Roberto Cánua)]

There are 241 threads of /usr/share/elasticsearch/jdk/bin/java and this is consumig
the RAM progressively. The ingestion process is executed every hour.

root@SRV-ELK:/etc/elasticsearch# ps -feaL | grep '/usr/share/elasticsearch/jdk/bin/java' | wc -l
241

The VM has 8 processors, 38GB RAM.
/etc/elasticsearch/elasticsearch.yml has the following sentences.

indices.queries.cache.size: 10%
indices.fielddata.cache.size: 10%

The jvm.options:

-Xms10g
-Xmx14g

When I stop the instances of elasticsearch and kibana,also execute the following command

sudo sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

This clean the RAM Cache and Buffers, but almost immediately start to increase consuming the RAM.

What am I missing.?
Thanks in advance.
(See Image attached from Grafana).

[RobertC1 (Roberto Cánua)]

This is another hint that the elasticsearch is the problem

root@SRV-ELK:/opt/trabaja/csv# ps aux --sort=-%mem | head -n 11
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
elastic+  831921  298 86.4 314512984 35460424 ?  Sl   13:05 219:57 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=org.elasticsearch.preallocate -Xms10g -Xmx14g -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9783747863161313092 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms10g -Xmx14g -server -XX:MaxDirectMemorySize=7516192768 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25 -Des.distribution.type=deb --module-path /usr/share/elasticsearch/lib --add-modules=jdk.net --add-modules=org.elasticsearch.preallocate -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch

[leandrojmp (Leandro Pereira)]

Are you having any issue?

The image you shared show just around 14 GB of memory used and the majority of the remaining RAM is in the cache.

This looks like normal behavior for me.

For example, my hot nodes have 64 GB and 30 GB of heap, the majority of memory used is the HEAP memory and the rest will appear as cached in the monitoring.

It says that it has only 880 MB free, but if you check on the server with free -h you have this:

              total        used        free      shared  buff/cache   available
Mem:           62Gi        37Gi       1.1Gi        16Mi        23Gi        24Gi
Swap:            0B          0B          0B

So, in my case 24 GB is available, buffered/cached memory does not count as used.

What is the response of free -h in your server?

[RobertC1 (Roberto Cánua)]

              total       usado       libre  compartido búfer/caché  disponible
Memoria:        39Gi       5,4Gi       8,7Gi       4,0Mi        25Gi        33Gi
Swap:          15Gi       4,0Mi        15Gi

[leandrojmp (Leandro Pereira)]

Your system has 38 GB and it says that 33 GB are available, not sure what is the issue here.

Memory that shows up as buffered/cached will be used by the kernel if an application needs it.

A better explanation can be found here: https://www.linuxatemyram.com

[RobertC1 (Roberto Cánua)]

The last information that a shared, is not accurate. Give me 15 minutes.

[RobertC1 (Roberto Cánua)]

Thanks @leandrojmp
I will follow the articel shared.