# Not able to start elasticsearch due to failed memory lock

**URL:** <https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009>\
**Category:** Elasticsearch\
**Created:** [November 23, 2018, 12:34pm UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009 "2018-11-23T12:34:33Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![frasmd](https://avatars.discourse-cdn.com/v4/letter/f/919ad9/32.png) [@frasmd](https://discuss.elastic.co/u/frasmd)\
**Post date:** [November 23, 2018, 12:34pm UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/1 "2018-11-23T12:34:34Z")

</div>

I am trying to install elasticstash 6.2.4 using RPMs. When i was tring to start service i was was getting below errors my elasticsearch logs.

[2] bootstrap checks failed  
[1]: memory locking requested for elasticsearch process but memory is not locked  
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk  
[2018-11-23T04:18:07,472][INFO][o.e.n.Node] [EjBNp8o] stopping ...  
[2018-11-23T04:18:07,607][INFO][o.e.n.Node] [EjBNp8o] stopped  
[2018-11-23T04:18:07,608][INFO][o.e.n.Node] [EjBNp8o] closing ...  
[2018-11-23T04:18:07,614][INFO][o.e.n.Node] [EjBNp8o] closed

I have tried all the threads online but nothing got me out of this error.

My config files are below for your reference,

1. /etc/elasticsearch/elasticsearch.yml

```auto
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------'''

```

1. /etc/sysconfig/elasticsearch

```auto
# Elasticsearch configuration directory
ES_PATH_CONF=/etc/elasticsearch

# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS
ES_JAVA_OPTS="-Xms10g -Xmx10g"

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
# Elasticsearch service
################################

# SysV init.d
#
# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5

################################
# System properties
################################

# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65536

# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using systemd, LimitMEMLOCK must be set in a unit file such as
# /etc/systemd/system/elasticsearch.service.d/override.conf.
MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144

```

1. /etc/security/limits.conf

```auto
#@student - maxlogins 4
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
# End of file

```

1. /usr/lib/systemd/system/elasticsearch.service

```auto
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
LimitMEMLOCK=infinity

```

Did i miss something? Please help me out. We are planning to deploy this for our production, we want this for the POC of elasticsearch.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Magnus\_Kessler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnus_kessler/32/42001_2.png) [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)\
**Post date:** [November 23, 2018, 2:06pm UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/2 "2018-11-23T14:06:54Z")

</div>

What Linux distribution do you run Elasticsearch on? Older versions of RHEL/Centos did not support memory locking, IIRC.

Also, please use ``` around the content of configuration files, for better readability.

---

<div class="post-metadata">

**Author:** ![frasmd](https://avatars.discourse-cdn.com/v4/letter/f/919ad9/32.png) [@frasmd](https://discuss.elastic.co/u/frasmd)\
**Post date:** [November 24, 2018, 11:11am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/3 "2018-11-24T11:11:01Z")

</div>

I am using CentOS 7.4. On the same OS i installed 5.1.1. I was able to deploy it and memory locking worked on the same version of OS. We wanted to xpack so i came to 6.2.4 and here i am stuck with mlock. Something i need to enable.

---

<div class="post-metadata">

**Author:** ![Magnus\_Kessler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnus_kessler/32/42001_2.png) [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)\
**Post date:** [November 24, 2018, 1:08pm UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/4 "2018-11-24T13:08:48Z")

</div>

Thanks for fixing up the config file formatting in your original message.

What does the `elasticsearch.service` file look like? Have you just added the `Limit*` lines to the file or is this the entire file?

The [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html#systemd) recommends to create an `override.conf` file with `systemctl edit elasticsearch` with this content:

```auto
[Service]
LimitMEMLOCK=infinity

```

Other `Limit*` settings are already correctly configured in the distributed service file.

---

<div class="post-metadata">

**Author:** ![frasmd](https://avatars.discourse-cdn.com/v4/letter/f/919ad9/32.png) [@frasmd](https://discuss.elastic.co/u/frasmd)\
**Post date:** [November 26, 2018, 7:45am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/5 "2018-11-26T07:45:57Z")

</div>

Yes i have made changes to the file you mentioned

```auto
[Service]
LimitMEMLOCK=infinity

```

Even then i get the following error on the logs and service wouldn't start

```auto
[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-11-23T04:18:07,472][INFO][o.e.n.Node] [EjBNp8o] stopping ...
[2018-11-23T04:18:07,607][INFO][o.e.n.Node] [EjBNp8o] stopped
[2018-11-23T04:18:07,608][INFO][o.e.n.Node] [EjBNp8o] closing ...
[2018-11-23T04:18:07,614][INFO][o.e.n.Node] [EjBNp8o] closed

```

```auto
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Mon 2018-11-26 01:45:26 CST; 3s ago
     Docs: http://www.elastic.co
  Process: 249655 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 249655 (code=exited, status=1/FAILURE)

Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: Started Elasticsearch.
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: Starting Elasticsearch...
Nov 26 01:45:26 er804centodell.mavdallab.com elasticsearch[249655]: Invalid initial heap size: -Xms10g
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: Unit elasticsearch.service entered failed state.
Nov 26 01:45:26 er804centodell.mavdallab.com systemd[1]: elasticsearch.service failed.

```

Am i missing something?

---

<div class="post-metadata">

**Author:** ![Magnus\_Kessler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnus_kessler/32/42001_2.png) [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)\
**Post date:** [November 26, 2018, 8:02am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/6 "2018-11-26T08:02:27Z")

</div>

Have a look at this previous [post](https://discuss.elastic.co/t/es-5-2-node-validation-exception-bootstrap-checks-failed-no-other-error/77922/4), which described a similar issue.

Could you please check if the logs contain messages that show in more detail why the syscall filters failed? It may have to do with the way the kernel was compiled.

If all else fails, you may have to disable memlock and syscall filtering (see the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/_system_call_filter_check.html), on how to do this).

---

<div class="post-metadata">

**Author:** ![frasmd](https://avatars.discourse-cdn.com/v4/letter/f/919ad9/32.png) [@frasmd](https://discuss.elastic.co/u/frasmd)\
**Post date:** [November 26, 2018, 9:28am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/7 "2018-11-26T09:28:47Z")

</div>

This is full error i get when i try to start the service

```auto
java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-x86/libjnidispatch.so) not found in resource path ([file:/usr/share/elasticsearch/lib/HdrHistogram-2.1.9.jar, file:/usr/share/elasticsearch/lib/elasticsearch-6.2.4.jar, file:/usr/share/elasticsearch/lib/elasticsearch-cli-6.2.4.jar, file:/usr/share/elasticsearch/lib/elasticsearch-core-6.2.4.jar, file:/usr/share/elasticsearch/lib/elasticsearch-launchers-6.2.4.jar, file:/usr/share/elasticsearch/lib/hppc-0.7.1.jar, file:/usr/share/elasticsearch/lib/jackson-core-2.8.10.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.10.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.8.10.jar, file:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.10.jar, file:/usr/share/elasticsearch/lib/jna-4.5.1.jar, file:/usr/share/elasticsearch/lib/joda-time-2.9.9.jar, file:/usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar, file:/usr/share/elasticsearch/lib/jts-1.13.jar, file:/usr/share/elasticsearch/lib/log4j-1.2-api-2.9.1.jar, file:/usr/share/elasticsearch/lib/log4j-api-2.9.1.jar, file:/usr/share/elasticsearch/lib/log4j-core-2.9.1.jar, file:/usr/share/elasticsearch/lib/lucene-analyzers-common-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-backward-codecs-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-core-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-grouping-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-highlighter-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-join-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-memory-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-misc-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-queries-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-queryparser-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-sandbox-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-spatial-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-spatial-extras-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-spatial3d-7.2.1.jar, file:/usr/share/elasticsearch/lib/lucene-suggest-7.2.1.jar, file:/usr/share/elasticsearch/lib/plugin-classloader-6.2.4.jar, file:/usr/share/elasticsearch/lib/plugin-cli-6.2.4.jar, file:/usr/share/elasticsearch/lib/securesm-1.2.jar, file:/usr/share/elasticsearch/lib/snakeyaml-1.17.jar, file:/usr/share/elasticsearch/lib/spatial4j-0.6.jar, file:/usr/share/elasticsearch/lib/t-digest-3.0.jar])
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:962) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_181]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_181]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.4.jar:6.2.4]
[2018-11-26T03:23:11,629][WARN][o.e.b.Natives] cannot check if running as root because JNA is not available
[2018-11-26T03:23:11,629][WARN][o.e.b.Natives] cannot install system call filter because JNA is not available
[2018-11-26T03:23:11,629][WARN][o.e.b.Natives] cannot mlockall because JNA is not available
[2018-11-26T03:23:11,642][WARN][o.e.b.Natives] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2018-11-26T03:23:11,822][INFO][o.e.n.Node] [] initializing ...
[2018-11-26T03:23:11,894][INFO][o.e.e.NodeEnvironment] [EjBNp8o] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [550.3gb], net total_space [552.9gb], types [rootfs]
[2018-11-26T03:23:11,894][INFO][o.e.e.NodeEnvironment] [EjBNp8o] heap size [1.9gb], compressed ordinary object pointers [unknown]
[2018-11-26T03:23:11,895][INFO][o.e.n.Node] node name [EjBNp8o] derived from node ID [EjBNp8omTYmNPc3PusDUoQ]; set [node.name] to override
[2018-11-26T03:23:11,896][INFO][o.e.n.Node] version[6.2.4], pid[256026], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/i386], JVM[Oracle Corporation/Java HotSpot(TM) Server VM/1.8.0_181/25.181-b13]
[2018-11-26T03:23:11,896][INFO][o.e.n.Node] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.39xO4CeK, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -
[2018-11-26T03:23:15,479][INFO][o.e.n.Node] initialized
[2018-11-26T03:23:15,479][INFO][o.e.n.Node] [EjBNp8o] starting ...
[2018-11-26T03:23:15,842][INFO][o.e.t.TransportService] [EjBNp8o] publish_address 
[2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-11-26T03:23:15,997][INFO][o.e.n.Node] [EjBNp8o] closing ...
[2018-11-26T03:23:16,014][INFO][o.e.n.Node] [EjBNp8o] closed

```

---

<div class="post-metadata">

**Author:** ![frasmd](https://avatars.discourse-cdn.com/v4/letter/f/919ad9/32.png) [@frasmd](https://discuss.elastic.co/u/frasmd)\
**Post date:** [November 26, 2018, 9:29am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/8 "2018-11-26T09:29:54Z")

</div>

This the full log (`/var/log/elasticsearch/elasticsearch.log`) i get when i try starting the service

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [November 26, 2018, 10:12am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/9 "2018-11-26T10:12:10Z")

</div>

> [@frasmd](#):
>
> java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-x86/libjnidispatch.so) not found in resource path

This indicates you’re using an x86 32-bit platform either on 32-bit hardware, a 32-bit OS, or a 32-bit JVM. We [do not support 32-bit](https://github.com/elastic/jna-build/blob/6c814a76658d01c094015c3c29acde6e6f794268/build.gradle#L72). You need to use x64 for full support.

---

<div class="post-metadata">

**Author:** ![Magnus\_Kessler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnus_kessler/32/42001_2.png) [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)\
**Post date:** [November 26, 2018, 10:34am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/10 "2018-11-26T10:34:00Z")

</div>

Thanks for sending the full logs. It appears that the bootstrap checks fail because of JNA not being available. This can happen if the `/tmp` directory is mounted with `noexec`. See the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html) for more details.

---

<div class="post-metadata">

**Author:** ![frasmd](https://avatars.discourse-cdn.com/v4/letter/f/919ad9/32.png) [@frasmd](https://discuss.elastic.co/u/frasmd)\
**Post date:** [November 26, 2018, 10:40am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/11 "2018-11-26T10:40:53Z")

</div>

Hey perfect that was the issue i guess, somehow 32-bit version of JVM was installed on the machine. I removed and installed 64-bit JVM and it's working like charm now. Again i got error about already set heap size not equal to max heap size. I changed to that and now elasticsearch started working.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [November 26, 2018, 11:13am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/12 "2018-11-26T11:13:19Z")

</div>

> [@frasmd](#):
>
> I removed and installed 64-bit JVM and it's working like charm now.

Yay!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 24, 2018, 11:13am UTC](https://discuss.elastic.co/t/not-able-to-start-elasticsearch-due-to-failed-memory-lock/158009/13 "2018-12-24T11:13:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.