Elasticserach 7.2 Unable to create cluster with error "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"

Elastic Stack Elasticsearch
1

Hello!
Standalone (single-node) Elasticsearch is work fine.

I tried creating one node cluster, but it results into error:

elasticsearch_1  | {"type": "server", "timestamp": "2020-10-14T11:38:41,989+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "template742.local",  "message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-10-14T11:38:41,993+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "template742.local",  "message": "This can result in part of the JVM being swapped out."  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-10-14T11:38:41,993+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "template742.local",  "message": "Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-10-14T11:38:41,993+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "template742.local",  "message": "These can be adjusted by modifying /etc/security/limits.conf, for example: \n\t# allow user 'elasticsearch' mlockall\n\telasticsearch soft memlock unlimited\n\telasticsearch hard memlock unlimited"  }
elasticsearch_1  | {"type": "server", "timestamp": "2020-10-14T11:38:41,994+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "template742.local",  "message": "If you are logged in interactively, you will have to re-login for the new limits to take effect."  }

Tell me please, is there some memory requirements for Elastic cluster?

Elasticsearch.yml:

cluster.name: "elk-cluster"
network.host: 0.0.0.0
node.name: primary

discovery.seed_hosts:
  - 172.21.21.70
  - 172.21.21.75

path.repo: ['/usr/share/elasticsearch/backup']

bootstrap.memory_lock: true

xpack.license.self_generated.type: basic

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elasticsearch/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elasticsearch/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.http.ssl.verification_mode: none
2

All logs:

elasticsearch_1      | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,566+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,569+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "This can result in part of the JVM being swapped out."  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,570+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,572+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "These can be adjusted by modifying /etc/security/limits.conf, for example: \n\t# allow user 'elasticsearch' mlockall\n\telasticsearch soft memlock unlimited\n\telasticsearch hard memlock unlimited"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,573+0000", "level": "WARN", "component": "o.e.b.JNANatives", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "If you are logged in interactively, you will have to re-login for the new limits to take effect."  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,903+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [397.9gb], net total_space [886.1gb], types [ext4]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:08,907+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "heap size [15.9gb], compressed ordinary object pointers [true]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:09,718+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "node name [primary], node ID [RqQaxTmnSre0YJbou0RPLA], cluster name [elk-cluster]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:09,719+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/3.16.0-6-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:09,720+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "JVM home [/usr/share/elasticsearch/jdk]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:09,720+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11247197651105664652, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xmx16g, -Xms16g, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=8589934592, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,731+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [aggs-matrix-stats]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,735+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [analysis-common]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,735+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [data-frame]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,736+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [ingest-common]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,736+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [ingest-geoip]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,736+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [ingest-user-agent]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,737+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [lang-expression]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,745+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [lang-mustache]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,745+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [lang-painless]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,746+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [mapper-extras]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,746+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [parent-join]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,746+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [percolator]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,746+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [rank-eval]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,747+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [reindex]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,747+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [repository-url]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,747+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [transport-netty4]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,748+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-ccr]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,748+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-core]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,748+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-deprecation]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,749+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-graph]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,749+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-ilm]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,749+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-logstash]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,750+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-ml]"  }
3 
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,750+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-monitoring]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,750+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-rollup]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,751+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-security]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,751+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-sql]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,751+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "loaded module [x-pack-watcher]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:12,754+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "no plugins loaded"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:19,380+0000", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:20,166+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "[controller/108] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:21,024+0000", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:22,636+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "using discovery type [zen] and seed hosts providers [settings]"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:23,738+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "initialized"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:23,739+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "starting ..."  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,235+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "publish_address {172.19.0.2:9300}, bound_addresses {0.0.0.0:9300}"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,245+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks"  }
elasticsearch_1      | ERROR: [1] bootstrap checks failed
elasticsearch_1      | [1]: memory locking requested for elasticsearch process but memory is not locked
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,259+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "stopping ..."  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,288+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "stopped"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,288+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "closing ..."  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,312+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "closed"  }
elasticsearch_1      | {"type": "server", "timestamp": "2020-10-14T13:34:24,315+0000", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "elk-cluster", "node.name": "primary",  "message": "Native controller process has stopped - no new native processes can be started"  }
database_elasticsearch_1 exited with code 78
4

The problem is that you have set bootstrap.memory_lock: true but your system does not support memory locking. The solution is therefore not to set bootstrap.memory_lock: true. Instead you should disable swap.

2 Likes
5

Thnaks David! You are genius!
Can you explain me, why standalone Elastic work fine (same configuration)?

6

I expect you were running your one-node cluster in development mode which doesn't enforce this kind of production-readiness check.

1 Like
7

Yes, use discovery.type: single-node.
Thanks again!!)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.