# Elasticsearch Process getting killed **URL:** \ **Category:** Elasticsearch\ **Created:** [October 29, 2019, 2:35pm UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691 "2019-10-29T14:35:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![akshaymaniyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshaymaniyar/32/45836_2.png) [@akshaymaniyar](https://discuss.elastic.co/u/akshaymaniyar)\ **Post date:** [October 29, 2019, 2:35pm UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/1 "2019-10-29T14:35:18Z")
Recently we have started seeing that our elastic-search process is getting killed. We are using 5-5-1 version of elasticsearch. Any idea what could be the cause? ``` ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled) Drop-In: /etc/systemd/system/elasticsearch.service.d └─override.conf Active: failed (Result: signal) since Tue 2019-10-29 16:25:50 IST; 3h 31min ago Docs: http://www.elastic.co Process: 18823 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=killed, signal=KILL) Process: 18820 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS) Main PID: 18823 (code=killed, signal=KILL) ``` Below are the kernel logs: > ---
**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\ **Post date:** [October 30, 2019, 11:37am UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/2 "2019-10-30T11:37:31Z")
Hi @akshaymaniyar See these two lines from the logs you pasted: ```auto Oct 29 16:25:48 cms-zulu-datastore-none-1551817 kernel: [103329.931375] Out of memory: Kill process 18823 (java) score 830 or sacrifice child Oct 29 16:25:48 cms-zulu-datastore-none-1551817 kernel: [103329.932556] Killed process 18823 (java) total-vm:241920696kB, anon-rss:18411644kB, file-rss:26660112kB, shmem-rss:0kB ``` The system killed the ES process because it was using too much memory. You'll have to either reduce the amount of memory used by ES (setting smaller heap size via jvm.options), free up memory on the system by other means or (relatively unlikely to be broken unless you made changes to it) fix your system's settings to make the OOM killer less trigger happy. ---
**Author:** ![akshaymaniyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshaymaniyar/32/45836_2.png) [@akshaymaniyar](https://discuss.elastic.co/u/akshaymaniyar)\ **Post date:** [October 31, 2019, 6:22am UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/3 "2019-10-31T06:22:12Z")
We are running elasticsearch on a machine which has 52 GB ram. We are just running elasticsearch on this machine and have allotted 16GB of heap space. Below is the elasticsearch process: ``` elastic+ 5727 179 77.2 247020628 42213928 ? SLsl Oct29 4075:47 /usr/bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch elastic+ 5845 0.0 0.0 131304 7988 ? Sl Oct29 0:00 /usr/share/elasticsearch/plugins/x-pack/platform/linux-x86_64/bin/controller ``` Below is the usual memory usage of the machine ``` total used free shared buff/cache available Mem: 52 18 1 0 32 33 Swap: 0 0 0 ``` Are we missing anything? ---
**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\ **Post date:** [October 31, 2019, 10:13am UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/4 "2019-10-31T10:13:15Z")
This would imply an issue with your system settings from the looks of it. What are your system's settings for `vm .overcommit_memory` and `vm.swappiness`? ---
**Author:** ![akshaymaniyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshaymaniyar/32/45836_2.png) [@akshaymaniyar](https://discuss.elastic.co/u/akshaymaniyar)\ **Post date:** [November 3, 2019, 1:33pm UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/5 "2019-11-03T13:33:50Z")
Below are the values: ``` cat /proc/sys/vm/overcommit_memory 0 cat /proc/sys/vm/swappiness 60 ``` What are the recommended ones for a machine running elasticsearch ---
**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\ **Post date:** [November 3, 2019, 1:50pm UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/6 "2019-11-03T13:50:40Z")
We do not make any recommendations on `overcommit_memory` as far as I know since the correct setting here is somewhat use-case and system specific. We do however recommend turning swapping off or at least down significantly (see [https://www.elastic.co/guide/en/elasticsearch/reference/5.5/setup-configuration-memory.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/setup-configuration-memory.html)). I would recommend moving to a `swappiness` of `1` as per the linked docs as a value of `60` likely is detrimental to the performance of your ES process because of the risk of random swapping. This is however not necessarily going to fix the out of memory killer killing the ES process. Given that your system has significantly more memory available than outright needed to run ES with a 16GB heap you could try fixing the issue by allowing the system to allocate more aggressively via ```auto echo 1 > /proc/sys/vm/overcommit_memory ``` EDIT: One other thing you should look into is this line: ```auto Oct 29 16:25:48 cms-zulu-datastore-none-1551817 kernel: [103314.853172] INFO: task java:29057 blocked for more than 120 seconds. ``` It seems your `java` process became completely blocked on "something" here. That "something" is most likely disk IO. Could there be a problem with that such that disk IO resources are temporarily almost completely exhausted? What kind of storage hardware do you have backing your cluster? ---
**Author:** ![akshaymaniyar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshaymaniyar/32/45836_2.png) [@akshaymaniyar](https://discuss.elastic.co/u/akshaymaniyar)\ **Post date:** [November 4, 2019, 3:51am UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/7 "2019-11-04T03:51:22Z")
Regarding swappiness, we already have the below setting set in elasticsearch.yml `bootstrap.memory_lock: true` Do you recommend to do both =\> change the swappiness to 1 and bootstrap.memory\_lock: true Will get back on the IO metrics and the kind of hardware being used. ---
**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\ **Post date:** [November 4, 2019, 11:57pm UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/8 "2019-11-04T23:57:33Z")
> [@akshaymaniyar](#): > > Do you recommend to do both =\> change the swappiness to 1 and bootstrap.memory\_lock: true Yes, unrelated to this problem it's still a good idea to turn down swappiness even with `mlock` in place as `mlock` does not extend to ES' off-heap memory use. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 3, 2019, 12:05am UTC](https://discuss.elastic.co/t/elasticsearch-process-getting-killed/205691/9 "2019-12-03T00:05:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.