Hi All,
I am facing a serious error in my elasticsearch instance. Before I start explaining what is the error. I want to give a short info about current ecosystem i am playing with.
So we have a test server(LINUX) with 3 GB of RAM available with 2019 MB almost always freely available. And 1999 MB of Swap memory with 1383 MB freely available.
I have 2 instances of Elasticsearch. Both are version 6.2.4. The only difference is one is secured with Search Guard and other is not TLS secured. Now until recently for some reason the unsecured version( usually used for experiment and a lot of useless sample indexes) is unable to start. And whenever I try to start it, it produces "hs_err_pid12345.log". The log says that there is insufficient memory for JAVA RUNTIME ENV. Also ES logs keep on saying GS Allocation Failed.
Since its quite long, the content of "hs_err_pid12345.log" is at the end..
The other secured version used o work fine. But recently it has started to go down automatically. I am assuming it is facing similar issue?
Now I have tried various methods to resolve it.
- I tried allocating 2 GB of Max and Min heap in jvm.options(didn't work)
- I read somewhere that above only makes difference while running ES for first time, so I created env variable as mentioned in documentation.(didn't work)
- I tried shutting down ES secured version, maybe it was consuming some memory(didn't work).
- Deleted unnecessary Physical/ROM memory drive and changes it from 1.8 to 2.3 GB(didn't work)
Finally I wanted to delete unnecessary indices from unsecured ES. But i cannot put it live even once to do that.
Finally, what are my options?
How can I make it work?
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 4207738880 bytes for committing reserved memory.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.Out of Memory Error (os_linux.cpp:2640), pid=27501, tid=0x00007f40da7b1700
JRE version: (8.0_161-b12) (build )
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again--------------- T H R E A D ---------------
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x00007f40d400b800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=27539, stack(0x00007f40da6b2000,0x00007f40da7b2000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No eventsDeoptimization events (0 events):
No eventsClasses redefined (0 events):
No eventsInternal exceptions (0 events):
No eventsEvents (0 events):
No eventsEnvironment Variables:
JAVA_HOME=/abcd/username/jdk1.8.0_161/--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 7.6 (Maipo)
uname:Linux 3.10.0-957.12.2.el7.x86_64 #1 SMP Fri Apr 19 21:09:07 UTC 2019 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 65536, NOFILE 65536, AS infinity
load average:0.08 0.04 0.05/proc/meminfo:
MemTotal: 3880944 kB
MemFree: 105132 kB
MemAvailable: 190676 kB
Buffers: 0 kB
Cached: 269236 kB
SwapCached: 30552 kB
Active: 2565312 kB
Inactive: 1008092 kB
Active(anon): 2433264 kB
Inactive(anon): 875208 kB
Active(file): 132048 kB
Inactive(file): 132884 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2047996 kB
SwapFree: 1627132 kB
Dirty: 80 kB
Writeback: 0 kB
AnonPages: 3279516 kB
Mapped: 43808 kB
Shmem: 4300 kB
Slab: 84012 kB
SReclaimable: 50760 kB
SUnreclaim: 33252 kB
KernelStack: 9152 kB
PageTables: 16032 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3988468 kB
Committed_AS: 5002496 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 163068 kB
VmallocChunk: 34359341052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 585728 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 75712 kB
DirectMap2M: 3069952 kB
DirectMap1G: 3145728 kBCPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 85 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, 3dnowpref, tsc, tscinvbit
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
stepping : 4
microcode : 0x200004d
cpu MHz : 2693.671
cache size : 33792 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ssbd ibrs ibpb stibp fsgsbase smep arat spec_ctrl intel_stibp flush_l1d arch_capabilities
bogomips : 5387.34
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:Memory: 4k page, physical 3880944k(105132k free), swap 2047996k(1627132k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.161-b12) for linux-amd64 JRE (1.8.0_161-b12), built on Dec 19 2017 16:12:43 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Fri Aug 30 11:17:30 2019
elapsed time: 0 seconds (0d 0h 0m 0s)