Not able to start elasticsearch due to failed memory lock

frasmd · November 23, 2018, 12:34pm

I am trying to install elasticstash 6.2.4 using RPMs. When i was tring to start service i was was getting below errors my elasticsearch logs.

[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-11-23T04:18:07,472][INFO ][o.e.n.Node ] [EjBNp8o] stopping ...
[2018-11-23T04:18:07,607][INFO ][o.e.n.Node ] [EjBNp8o] stopped
[2018-11-23T04:18:07,608][INFO ][o.e.n.Node ] [EjBNp8o] closing ...
[2018-11-23T04:18:07,614][INFO ][o.e.n.Node ] [EjBNp8o] closed

I have tried all the threads online but nothing got me out of this error.

My config files are below for your reference,

/etc/elasticsearch/elasticsearch.yml

#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------'''

/etc/sysconfig/elasticsearch

# Elasticsearch configuration directory
ES_PATH_CONF=/etc/elasticsearch

# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS
ES_JAVA_OPTS="-Xms10g -Xmx10g"

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
# Elasticsearch service
################################

# SysV init.d
#
# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5

################################
# System properties
################################

# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65536

# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using systemd, LimitMEMLOCK must be set in a unit file such as
# /etc/systemd/system/elasticsearch.service.d/override.conf.
MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144

/etc/security/limits.conf

#@student        -       maxlogins       4
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
# End of file

/usr/lib/systemd/system/elasticsearch.service

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
LimitMEMLOCK=infinity

Did i miss something? Please help me out. We are planning to deploy this for our production, we want this for the POC of elasticsearch.

Thanks in advance.

Magnus_Kessler · November 23, 2018, 2:06pm

What Linux distribution do you run Elasticsearch on? Older versions of RHEL/Centos did not support memory locking, IIRC.

Also, please use ``` around the content of configuration files, for better readability.

frasmd · November 24, 2018, 11:11am

I am using CentOS 7.4. On the same OS i installed 5.1.1. I was able to deploy it and memory locking worked on the same version of OS. We wanted to xpack so i came to 6.2.4 and here i am stuck with mlock. Something i need to enable.

Magnus_Kessler · November 24, 2018, 1:08pm

Thanks for fixing up the config file formatting in your original message.

What does the elasticsearch.service file look like? Have you just added the Limit* lines to the file or is this the entire file?

The documentation recommends to create an override.conf file with systemctl edit elasticsearch with this content:

[Service]
LimitMEMLOCK=infinity

Other Limit* settings are already correctly configured in the distributed service file.

frasmd · November 26, 2018, 7:45am

Yes i have made changes to the file you mentioned

[Service]
LimitMEMLOCK=infinity

Even then i get the following error on the logs and service wouldn't start

[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-11-23T04:18:07,472][INFO ][o.e.n.Node               ] [EjBNp8o] stopping ...
[2018-11-23T04:18:07,607][INFO ][o.e.n.Node               ] [EjBNp8o] stopped
[2018-11-23T04:18:07,608][INFO ][o.e.n.Node               ] [EjBNp8o] closing ...
[2018-11-23T04:18:07,614][INFO ][o.e.n.Node               ] [EjBNp8o] closed

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Mon 2018-11-26 01:45:26 CST; 3s ago
     Docs: http://www.elastic.co
  Process: 249655 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 249655 (code=exited, status=1/FAILURE)

Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: Started Elasticsearch.
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: Starting Elasticsearch...
Nov 26 01:45:26 er804centodell.mavdallab.com elasticsearch[249655]: Invalid initial heap size: -Xms10g
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: Unit elasticsearch.service entered failed state.
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: elasticsearch.service failed.

Am i missing something?

Magnus_Kessler · November 26, 2018, 8:02am

Have a look at this previous post, which described a similar issue.

Could you please check if the logs contain messages that show in more detail why the syscall filters failed? It may have to do with the way the kernel was compiled.

If all else fails, you may have to disable memlock and syscall filtering (see the documentation, on how to do this).

frasmd · November 26, 2018, 9:28am

This is full error i get when i try to start the service

java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-x86/libjnidispatch.so) not found in resource path ([file:/usr/share/elasticsearch/lib/HdrHistogram-2.1.9.jar, file:/usr/share/elasticsearch/lib/elasticsearch-6.2.4.jar, file:/usr/share/elasticsearch/lib/elasticsearch-cli-6.2.4.jar, file:/usr/share/elasticsearch/lib/elasticsearch-core-6.2.4.jar, file:/usr/share/elasticsearch/lib/elasticsearch-launchers-6.2.4.jar, file:/usr/share/elasticsearch/lib/hppc-0.7.1.jar, file:/usr/share/elasticsearch/lib/jackson-core-2.8.10.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.10.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.8.10.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.10.jar, file:/usr/share/elasticsearch/lib/jna-4.5.1.jar, file:/usr/share/elasticsearch/lib/joda-time-2.9.9.jar, file:/usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar, file:/usr/share/elasticsearch/lib/jts-1.13.jar, file:/usr/share/elasticsearch/lib/log4j-1.2-api-2.9.1.jar, file:/usr/share/elasticsearch/lib/log4j-api-2.9.1.jar, file:/usr/share/elasticsearch/lib/log4j-core-2.9.1.jar, file:/usr/share/elasticsearch/lib/lucene-analyzers-common-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-backward-codecs-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-core-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-grouping-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-highlighter-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-join-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-memory-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-misc-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-queries-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-queryparser-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-sandbox-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-spatial-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-spatial-extras-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-spatial3d-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-suggest-7.2.1.jar, file:/usr/share/elasticsearch/lib/plugin-classloader-6.2.4.jar, file:/usr/share/elasticsearch/lib/plugin-cli-6.2.4.jar, file:/usr/share/elasticsearch/lib/securesm-1.2.jar, file:/usr/share/elasticsearch/lib/snakeyaml-1.17.jar, file:/usr/share/elasticsearch/lib/spatial4j-0.6.jar, file:/usr/share/elasticsearch/lib/t-digest-3.0.jar])
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:962) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_181]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_181]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
[2018-11-26T03:23:11,629][WARN ][o.e.b.Natives            ] cannot check if running as root because JNA is not available
[2018-11-26T03:23:11,629][WARN ][o.e.b.Natives            ] cannot install system call filter because JNA is not available
[2018-11-26T03:23:11,629][WARN ][o.e.b.Natives            ] cannot mlockall because JNA is not available
[2018-11-26T03:23:11,642][WARN ][o.e.b.Natives            ] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-11-26T03:23:11,822][INFO ][o.e.n.Node               ] [] initializing ...
[2018-11-26T03:23:11,894][INFO ][o.e.e.NodeEnvironment    ] [EjBNp8o] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [550.3gb], net total_space [552.9gb], types [rootfs]
[2018-11-26T03:23:11,894][INFO ][o.e.e.NodeEnvironment    ] [EjBNp8o] heap size [1.9gb], compressed ordinary object pointers [unknown]
[2018-11-26T03:23:11,895][INFO ][o.e.n.Node               ] node name [EjBNp8o] derived from node ID [EjBNp8omTYmNPc3PusDUoQ]; set [node.name] to override
[2018-11-26T03:23:11,896][INFO ][o.e.n.Node               ] version[6.2.4], pid[256026], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/i386], JVM[Oracle Corporation/Java HotSpot(TM) Server VM/1.8.0_181/25.181-b13]
[2018-11-26T03:23:11,896][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.39xO4CeK, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -
[2018-11-26T03:23:15,479][INFO ][o.e.n.Node               ] initialized
[2018-11-26T03:23:15,479][INFO ][o.e.n.Node               ] [EjBNp8o] starting ...
[2018-11-26T03:23:15,842][INFO ][o.e.t.TransportService   ] [EjBNp8o] publish_address 
[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-11-26T03:23:15,997][INFO ][o.e.n.Node               ] [EjBNp8o] closing ...
[2018-11-26T03:23:16,014][INFO ][o.e.n.Node               ] [EjBNp8o] closed

frasmd · November 26, 2018, 9:29am

This the full log (/var/log/elasticsearch/elasticsearch.log) i get when i try starting the service

jasontedor · November 26, 2018, 10:12am

This indicates you’re using an x86 32-bit platform either on 32-bit hardware, a 32-bit OS, or a 32-bit JVM. We do not support 32-bit. You need to use x64 for full support.

Magnus_Kessler · November 26, 2018, 10:34am

Thanks for sending the full logs. It appears that the bootstrap checks fail because of JNA not being available. This can happen if the /tmp directory is mounted with noexec. See the documentation for more details.

frasmd · November 26, 2018, 10:40am

Hey perfect that was the issue i guess, somehow 32-bit version of JVM was installed on the machine. I removed and installed 64-bit JVM and it's working like charm now. Again i got error about already set heap size not equal to max heap size. I changed to that and now elasticsearch started working.

jasontedor · November 26, 2018, 11:13am

Yay!

system · December 24, 2018, 11:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.