Hello,
Recently, we've had a server running elasticsearch randomly crashing without errors.
Upon further investigation we noticed it was crashing due to out-of-memory, as the following was in /var/log/kern.log:
[11533] 106 11533 18907505 8142258 17202 77 0 0 java
Out of memory: Kill process 11533 (java) score 247 or sacrifice child
Killed process 11533 (java) total-vm:75630020kB, anon-rss:32450404kB, file-rss:118628kB
My server settings are:
- The server has 128 GB ram
- Ubuntu 14.04 LTS
- No swap configured, memory lock enabled
- Heap size 31g
- ES Version: 2.4.4
- Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
The weird thing is, I've never seen the machine getting close to using all of its 128GB ram (according to top), so I don't know why the process is being killed.
Any ideas?