# Elasticsearch is not starting when bootstrap.memory\_lock is set to true

**URL:** <https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962>\
**Category:** Elasticsearch\
**Created:** [February 22, 2018, 1:59am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962 "2018-02-22T01:59:12Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 1:59am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/1 "2018-02-22T01:59:12Z")

</div>

Hi,

Elasticsearch throws following error when bootstrap.memory\_lock is set to true,

```
[2018-02-21T15:00:09,690][INFO][o.e.t.TransportService] [mynode] publish_address {xxxxxx:9300}, bound_addresses {[::]:9300}
[2018-02-21T15:00:09,700][INFO][o.e.b.BootstrapChecks] [mynode] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-02-21T15:00:09,701][ERROR][o.e.b.Bootstrap] [mynode] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2018-02-21T15:00:09,744][INFO][o.e.n.Node] [mynode] stopping ...
[2018-02-21T15:00:09,765][INFO][o.e.n.Node] [mynode] stopped
[2018-02-21T15:00:09,765][INFO][o.e.n.Node] [mynode] closing ...
[2018-02-21T15:00:09,773][INFO][o.e.n.Node] [mynode] closed
[2018-02-21T15:03:18,927][WARN][o.e.b.JNANatives] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-02-21T15:03:18,929][WARN][o.e.b.JNANatives] This can result in part of the JVM being swapped out.

```

Can someone please explain the problem?

Thanks

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [February 22, 2018, 2:39am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/2 "2018-02-22T02:39:38Z")

</div>

You requested memory locking but Elasticsearch was not able to fulfill your request. Rather than silently ignore the failure, we let you know. Higher in your logs you should see the reason why the memory locking failed. This is usually an environmental problem (e.g., the user running the Elasticsearch process does not have sufficient permissions to lock memory).

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 3:06am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/3 "2018-02-22T03:06:15Z")

</div>

Thanks for your quick response Jason.

I have gone through some more logs, please see if these can make sense,

```
[2018-02-21T18:46:35,853][WARN][o.e.b.JNANatives] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-02-21T18:46:35,855][WARN][o.e.b.JNANatives] This can result in part of the JVM being swapped out.
[2018-02-21T18:46:35,855][WARN][o.e.b.JNANatives] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-02-21T18:46:35,856][WARN][o.e.b.JNANatives] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2018-02-21T18:46:35,856][WARN][o.e.b.JNANatives] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-02-21T18:46:35,927][INFO][o.e.n.Node] [mynode] initializing ...
[2018-02-21T18:46:35,997][INFO][o.e.e.NodeEnvironment] [mynode] using [1] data paths, mounts [[/data (/dev/mapper/xxxxx)]], net usable_space [499.6gb], net total_space [499.7gb], spins? [possibly], types [xfs]
[2018-02-21T18:46:35,997][INFO][o.e.e.NodeEnvironment] [mynode] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-21T18:46:35,998][INFO][o.e.n.Node] [mynode] node name [mynode], node ID [xvye9atOTUSeEyigdrfg]
[2018-02-21T18:46:35,999][INFO][o.e.n.Node] [mynode] version[5.6.7], pid[23561], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.11.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]
@        
[2018-02-21T18:46:35,999][INFO][o.e.n.Node] [mynode] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-21T18:46:36,636][INFO][o.e.p.PluginsService] [mynode] loaded module [aggs-matrix-stats]
[2018-02-21T18:46:36,636][INFO][o.e.p.PluginsService] [mynode] loaded module [ingest-common]
[2018-02-21T18:46:36,636][INFO][o.e.p.PluginsService] [mynode] loaded module [lang-expression]
[2018-02-21T18:46:36,636][INFO][o.e.p.PluginsService] [mynode] loaded module [lang-groovy]
```

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [February 22, 2018, 3:17am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/4 "2018-02-22T03:17:49Z")

</div>

Your problem is right here:

> [@amruth](#):
>
> ```auto
> [2018-02-21T18:46:35,853][WARN][o.e.b.JNANatives] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
> [2018-02-21T18:46:35,855][WARN][o.e.b.JNANatives] This can result in part of the JVM being swapped out.
> [2018-02-21T18:46:35,855][WARN][o.e.b.JNANatives] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
> [2018-02-21T18:46:35,856][WARN][o.e.b.JNANatives] These can be adjusted by modifying /etc/security/limits.conf, for example:
> # allow user 'elasticsearch' mlockall
> elasticsearch soft memlock unlimited
> elasticsearch hard memlock unlimited
> 
> ```

As I mentioned, this is an environmental problem. How are you starting Elasticsearch? Sys V init, `systemd`, or directly from `bin/elasticsearch`?

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 3:21am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/5 "2018-02-22T03:21:27Z")

</div>

I am starting elasticsearch as a service in Redhat by using "sudo systemctl start elasticsearch.service "

What would be the problem here?

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 3:25am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/6 "2018-02-22T03:25:30Z")

</div>

To be more specific, I am using following commands -

sudo /bin/systemctl daemon-reload  
sudo /bin/systemctl enable elasticsearch.service  
sudo systemctl start elasticsearch.service

And I used the same exact procedure to setup Elasticsearch Cluster in a different environment and it runs perfectly fine without any issues. Only difference is the version - 5.5.1 in old environment and 5.6.7 in the current environment.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [February 22, 2018, 3:37am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/7 "2018-02-22T03:37:44Z")

</div>

The problem is that you need to enable memory locking as per the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/setting-system-settings.html#systemd). This gives the process the permissions to lock memory which it currently does not have.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 3:59am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/8 "2018-02-22T03:59:21Z")

</div>

This worked like charm. But I didn't do this kind of setting on the other Production server, why was there no issue? It's been running for 5 months without any issues.

Also is it mandatory to go through "Important System Configuration" section when setting up in Production? Settings I configured were, bootstarp.memorylock to true and jvm heap size to half the memory of VM.

Please suggest me if I am missing any important configurations.

Thanks

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 4:06am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/9 "2018-02-22T04:06:02Z")

</div>

My bad...Even on the old production server I see the setting "[Service]  
LimitMEMLOCK=infinity" in the file /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

But I don't remember configuring that setting. Would it be possible that it got configured as part of VM build out?

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [February 22, 2018, 4:13am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/10 "2018-02-22T04:13:04Z")

</div>

> [@amruth](#):
>
> Also is it mandatory to go through "Important System Configuration" section when setting up in Production?

No, but I encourage you to do this.

> [@amruth](#):
>
> Would it be possible that it got configured as part of VM build out?

That seems unlikely.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 4:14am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/11 "2018-02-22T04:14:45Z")

</div>

Got it. Really appreciate your help. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [February 22, 2018, 4:30am UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/12 "2018-02-22T04:30:41Z")

</div>

You're welcome.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 3:18pm UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/13 "2018-02-22T15:18:32Z")

</div>

One last question - Where to set JVM heap size? There are two locations in jvm.options file,

```
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
-Xms16g
-Xmx16g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

```

Can you please say where I need to set, the top one or the bottom one?

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [February 22, 2018, 3:29pm UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/14 "2018-02-22T15:29:21Z")

</div>

There are not two by default, the last one will win.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [February 22, 2018, 3:30pm UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/15 "2018-02-22T15:30:22Z")

</div>

Ok, thanks Jason

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [February 22, 2018, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/16 "2018-02-22T15:42:00Z")

</div>

In case it helps...

When deploying on RedHat/CentOS this is the file I always use at `/etc/systemd/system/elasticsearch.service.d/elasticsearch.conf`

```auto
[Service]
LimitMEMLOCK=infinity
LimitNPROC=4096
LimitNOFILE=65536

```

This covers the settings recommended in the documentation.

Rob

Robert Cowart ([rob@koiossian.com](mailto:rob@koiossian.com))  
[www.koiossian.com](http://www.koiossian.com)  
True Turnkey SOLUTIONS for the Elastic Stack

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2018, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-is-not-starting-when-bootstrap-memory-lock-is-set-to-true/120962/17 "2018-03-22T15:42:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.