Elastic search data node java process is consuming high virtual memory

Elastic Stack Elasticsearch
,
1

Hi Team,

we have elastic data node which is triggering alerts for high virtual memory , upon checking i could see below stats in server .

top - 07:25:19 up 549 days, 15 min, 4 users, load average: 0.76, 0.88, 0.90
Tasks: 373 total, 2 running, 370 sleeping, 0 stopped, 1 zombie
%Cpu(s): 6.9 us, 0.5 sy, 0.0 ni, 92.3 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32680152 total, 322644 free, 26089360 used, 6268148 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 12166028 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27684 elastic+ 20 0 420.3g 18.7g 918092 S 116.6 60.0 1225:35 java
3483 root 20 0 1851796 17104 2432 S 0.0 0.1 32:18.03 nsrexecd
1402 root 20 0 1606604 9200 6432 S 0.0 0.0 18:51.67 rsyslogd
1399 root 20 0 1546900 39028 1432 S 0.0 0.1 659:27.46 dockerd
1465 root 20 0 1245960 16316 940 S 0.0 0.0 1233:56 docker-containe
20243 root 20 0 1205852 10868 708 S 0.0 0.0 642:12.79 falcon-sensor
1404 root 20 0 553704 10700 0 S 0.0 0.0 70:51.89 tuned
1148 polkitd 20 0 534560 6968 0 S 0.0 0.0 28:45.88 polkitd
740 root 20 0 473756 4380 680 S 0.0 0.0 0:25.52 lvmetad

JVM allocation in our server

Xms represents the initial size of total heap space
Xmx represents the maximum size of total heap space

-Xms16g
-Xmx16g

Memory information :
MemTotal: 32680152 kB
MemFree: 266612 kB
MemAvailable: 12162172 kB
Buffers: 1028 kB
Cached: 5051604 kB
SwapCached: 0 kB
Active: 25068084 kB
Inactive: 5702052 kB
Active(anon): 19442440 kB
Inactive(anon): 143992 kB
Active(file): 5625644 kB
Inactive(file): 5558060 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 172704 kB
Writeback: 0 kB
AnonPages: 19564384 kB
Mapped: 1518900 kB
Shmem: 22048 kB
Slab: 1229088 kB
SReclaimable: 1012552 kB
SUnreclaim: 216536 kB
KernelStack: 14768 kB
PageTables: 193808 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 16340076 kB
Committed_AS: 20399940 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 216428 kB
VmallocChunk: 34359339004 kB
HardwareCorrupted: 0 kB
AnonHugePages: 366592 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 12517248 kB
DirectMap2M: 19988480 kB
DirectMap1G: 3145728 kB

Our monitoring tool service metrics :
Service Metrics active=25593479168;;;; active_anon=19892379648;;;; active_file=5701099520;;;; anon_huge_pages=375390208;;;; anon_pages=20016943104;;;; bounce=0;;;; buffers=4505600;;;; cached=4854808576;;;; caches=5900533760;;;; cma_free=0;;;; cma_total=0;;;; commit_limit=16732237824;;;; committed_as=20859682816;;;; dirty=165269504;;;; hardware_corrupted=0;;;; inactive=5756665856;;;; inactive_anon=147447808;;;; inactive_file=5609218048;;;; kernel_stack=15024128;;;; mapped=880381952;;;; mem_available=12464586752;;;; mem_free=420962304;;;; mem_total=33464475648;;;; mem_used=27142979584;;;; mlocked=0;;;; nfs_unstable=0;;;; page_tables=195354624;;;; pending=165273600;;;; sreclaimable=1041219584;;;; sunreclaim=221757440;;;; shmem=22577152;;;; slab=1262977024;;;; swap_cached=0;;;; swap_free=0;;;; swap_total=0;;;; swap_used=0;;;; total_total=33464475648;;;; total_used=27142979584;;;; unevictable=0;;;; writeback=4096;;;; writeback_tmp=0;;;;

Please let us know what might be triggering this alerts , please let us know do we need to change any thing in our server end .

Regards,
Krishna

2

Looks reasonable to me. Why is this a problem?

3

Thanks for looking into this @DavidTurner

I have set up threshold limit in our monitoring tool for warning 80 % critical 90%. which is triggering alerts do i need to change these threshold limits?

WARN - RAM used: 25.06 GB of 31.17 GB (80.4%), Total virtual memory used: 25.06 GB of 31.17 GB (80.4%, warn/crit at 80.0%/90.0%) WARN

any help would be appreciated.

4

You still haven't explained why this is a problem.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.