Elasticsearch is not starting when bootstrap.memory_lock is set to true

Elastic Stack Elasticsearch
1

Hi,

Elasticsearch throws following error when bootstrap.memory_lock is set to true,

[2018-02-21T15:00:09,690][INFO ][o.e.t.TransportService   ] [mynode] publish_address {xxxxxx:9300}, bound_addresses {[::]:9300}
[2018-02-21T15:00:09,700][INFO ][o.e.b.BootstrapChecks    ] [mynode] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-02-21T15:00:09,701][ERROR][o.e.b.Bootstrap          ] [mynode] node validation exception
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2018-02-21T15:00:09,744][INFO ][o.e.n.Node               ] [mynode] stopping ...
[2018-02-21T15:00:09,765][INFO ][o.e.n.Node               ] [mynode] stopped
[2018-02-21T15:00:09,765][INFO ][o.e.n.Node               ] [mynode] closing ...
[2018-02-21T15:00:09,773][INFO ][o.e.n.Node               ] [mynode] closed
[2018-02-21T15:03:18,927][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-02-21T15:03:18,929][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.

Can someone please explain the problem?

Thanks

2

You requested memory locking but Elasticsearch was not able to fulfill your request. Rather than silently ignore the failure, we let you know. Higher in your logs you should see the reason why the memory locking failed. This is usually an environmental problem (e.g., the user running the Elasticsearch process does not have sufficient permissions to lock memory).

3

Thanks for your quick response Jason.

I have gone through some more logs, please see if these can make sense,

[2018-02-21T18:46:35,853][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2018-02-21T18:46:35,855][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2018-02-21T18:46:35,855][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2018-02-21T18:46:35,856][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
        # allow user 'elasticsearch' mlockall
        elasticsearch soft memlock unlimited
        elasticsearch hard memlock unlimited
[2018-02-21T18:46:35,856][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2018-02-21T18:46:35,927][INFO ][o.e.n.Node               ] [mynode] initializing ...
[2018-02-21T18:46:35,997][INFO ][o.e.e.NodeEnvironment    ] [mynode] using [1] data paths, mounts [[/data (/dev/mapper/xxxxx)]], net usable_space [499.6gb], net total_space [499.7gb], spins? [possibly], types [xfs]
[2018-02-21T18:46:35,997][INFO ][o.e.e.NodeEnvironment    ] [mynode] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-02-21T18:46:35,998][INFO ][o.e.n.Node               ] [mynode] node name [mynode], node ID [xvye9atOTUSeEyigdrfg]
[2018-02-21T18:46:35,999][INFO ][o.e.n.Node               ] [mynode] version[5.6.7], pid[23561], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.11.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]
@        
[2018-02-21T18:46:35,999][INFO ][o.e.n.Node               ] [mynode] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-02-21T18:46:36,636][INFO ][o.e.p.PluginsService     ] [mynode] loaded module [aggs-matrix-stats]
[2018-02-21T18:46:36,636][INFO ][o.e.p.PluginsService     ] [mynode] loaded module [ingest-common]
[2018-02-21T18:46:36,636][INFO ][o.e.p.PluginsService     ] [mynode] loaded module [lang-expression]
[2018-02-21T18:46:36,636][INFO ][o.e.p.PluginsService     ] [mynode] loaded module [lang-groovy]
4

Your problem is right here:

As I mentioned, this is an environmental problem. How are you starting Elasticsearch? Sys V init, systemd, or directly from bin/elasticsearch?

5

I am starting elasticsearch as a service in Redhat by using "sudo systemctl start elasticsearch.service "

What would be the problem here?

6

To be more specific, I am using following commands -

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

And I used the same exact procedure to setup Elasticsearch Cluster in a different environment and it runs perfectly fine without any issues. Only difference is the version - 5.5.1 in old environment and 5.6.7 in the current environment.

7

The problem is that you need to enable memory locking as per the documentation. This gives the process the permissions to lock memory which it currently does not have.

3 Likes
8

This worked like charm. But I didn't do this kind of setting on the other Production server, why was there no issue? It's been running for 5 months without any issues.

Also is it mandatory to go through "Important System Configuration" section when setting up in Production? Settings I configured were, bootstarp.memorylock to true and jvm heap size to half the memory of VM.

Please suggest me if I am missing any important configurations.

Thanks

9

My bad...Even on the old production server I see the setting "[Service]
LimitMEMLOCK=infinity" in the file /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

But I don't remember configuring that setting. Would it be possible that it got configured as part of VM build out?

10

No, but I encourage you to do this.

That seems unlikely.

11

Got it. Really appreciate your help. Thanks a lot.

12

You're welcome.

13

One last question - Where to set JVM heap size? There are two locations in jvm.options file,

## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
-Xms16g
-Xmx16g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

Can you please say where I need to set, the top one or the bottom one?

14

There are not two by default, the last one will win.

1 Like
15

Ok, thanks Jason

16

In case it helps...

When deploying on RedHat/CentOS this is the file I always use at /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

[Service]
LimitMEMLOCK=infinity
LimitNPROC=4096
LimitNOFILE=65536

This covers the settings recommended in the documentation.

Rob

Robert Cowart (rob@koiossian.com)
www.koiossian.com
True Turnkey SOLUTIONS for the Elastic Stack

7 Likes
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.