Hi,
I have a dev server running elasticsearch 2.3.5, kibana 4.5 on ubuntu 16.04.1 LTS. Originally, this server has only 1G RAM, but it is constantly killed due to OOM. This forces me to add more memories to the VM. Now it has 4G memory, but it still get killed. I think something must be wrong with the settings because my production server which uses 4G memory never get OOM. Though the production server is ubuntu 14.04LTS. I am still confused about what's going wrong.
https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.x/setting-system-settings.html#sysconfig
https://www.elastic.co/guide/en/elasticsearch/reference/2.1/setup-configuration.html#setup-configuration-memory
According to the above links, I made the following configurations:
/etc/default/elasticsearch
ES_HEAP_SIZE=2g
ES_JAVA_OPTS="-Xms2g -Xmx2g"
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
/etc/elasticsearch/elasticsearch.yml
bootstrap.mlockall: true
/usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity
/etc/fstab
comment swap directly.
And I am still getting OOM kill messages in syslog everyday. It's very annoying to restart your elasticsearch service everyday. How could I fix this?
The followings are the OOM messages:
Nov 10 18:03:24 elk2dev kernel: [204872.040150] [ 1067] 0 1067 1431 884 8 3 0 -17 iscsid
Nov 10 18:03:24 elk2dev kernel: [204872.040152] [ 1121] 0 1121 4673 362 13 3 0 0 agetty
Nov 10 18:03:24 elk2dev kernel: [204872.040154] [ 1138] 0 1138 69272 178 39 3 0 0 polkitd
Nov 10 18:03:24 elk2dev kernel: [204872.040156] [ 3754] 111 3754 1059531 594141 1228 7 0 0 java
Nov 10 18:03:24 elk2dev kernel: [204872.040158] Out of memory: Kill process 3754 (java) score 588 or sacrifice child
Nov 10 18:03:24 elk2dev kernel: [204872.040285] Killed process 3754 (java) total-vm:4238124kB, anon-rss:2350724kB, file-rss:25840kB
Nov 10 18:03:25 elk2dev systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Nov 10 18:03:25 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:25+00:00","tags":["error","elasticsearch"],"pid":889,"message":"Request error, retrying -- connect ECONNREFUSED 127.0.0.1:9200"}
Nov 10 18:03:25 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:25+00:00","tags":["warning","elasticsearch"],"pid":889,"message":"Unable to revive connection: http://localhost:9200/"}
Nov 10 18:03:25 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:25+00:00","tags":["warning","elasticsearch"],"pid":889,"message":"No living connections"}
Nov 10 18:03:25 elk2dev systemd[1]: elasticsearch.service: Unit entered failed state.
Nov 10 18:03:25 elk2dev systemd[1]: elasticsearch.service: Failed with result 'signal'.
Nov 10 18:03:25 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:25+00:00","tags":["status","plugin:elasticsearch","error"],"pid":889,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
Nov 10 18:03:28 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:28+00:00","tags":["warning","elasticsearch"],"pid":889,"message":"Unable to revive connection: http://localhost:9200/"}
Nov 10 18:03:28 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:28+00:00","tags":["warning","elasticsearch"],"pid":889,"message":"No living connections"}
Nov 10 18:03:28 elk2dev kibana[889]: {"type":"log","@timestamp":"2016-11-10T10:03:28+00:00","tags":["status","plugin:elasticsearch","error"],"pid":889,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - Unable to connect to Elasticsearch at http://localhost:9200.","prevState":"red","prevMsg":"No Living connections"}