Elasticsearch service gets killed

Elastic Stack Elasticsearch
1

Hi,

I have a setup of ELK on one server. Elasticsearch service and kibana run fine until I start the logstash service.
When I start logstash, elasticsearch gets killed instantly and I see this error:

elasticsearch systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
elasticsearch systemd[1]: elasticsearch.service: Failed with result 'signal'.

I checked the journal logs and found

elasticsearch kernel: Out of memory: Kill process 5402 (java) score 556 or sacrifice child
elasticsearch kernel: Killed process 5402 (java) total-vm:12237240kB, anon-rss:8885908kB, file-rss:24300kB, shmem-rss:0kB

Some Configurations for reference:

cat /proc/sys/vm/swappiness
1
cat /proc/sys/vm/override
1
cat /proc/10056/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             4096                 4096                 processes 
Max open files            65536                65536                files     
Max locked memory         unlimited            unlimited            bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       62533                62533                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

What is making the system to kill the service??

Thanks and Regards
Hameed

2

This is linux OOM, so it means that you are running out of memory on this node. Add more RAM.

1 Like
3

I have 16GB RAM available on the server and have set the heap size as 8GB.
When I changed the heap size to 4GB, the elasticsearch doesn't get killed now.

What could be the reason this happens?

4

Hi i usually set heap to 1/3 of the machine RAM, that is the safe case.

5

Ok.

I had read to set to half the size which lead me to set 8GB, guess that's wrong then.
Will try and set it to 6GB and see what outcome I get.

Thanks for your help

6

Half of RAM is max i would set it, and no more than 32GB per machine.
I depends what else you have on that machine, if there is similar database( postgres also likes caching,etc).

7

How would I go about to calculate how much RAM is needed by elasticsearch for me?

8

That entirely based on your systems, i usually have script that reserves 30% of RAM to heap, and then i leave the rest for the system and other programs.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.