# Issues with jvm lock

**URL:** <https://discuss.elastic.co/t/issues-with-jvm-lock/291318>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [December 9, 2021, 11:37am UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318 "2021-12-09T11:37:25Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [December 9, 2021, 11:37am UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/1 "2021-12-09T11:37:25Z")

</div>

Hello,  
I have below issues,  
after start Elasticsearch I see warnings

```auto
"message": "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"
"message": "This can result in part of the JVM being swapped out."
"message": "memory locking requested for elasticsearch process but memory is not locked"

```

I'm using Elasticsearch in docker.  
version of ES is 7.11.1

I have issues that after some time of working with logstash I see errors in logstash that bulk request failed and probably ES is down for logstash.  
I think this is related with error of lock jvm memory

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [December 9, 2021, 8:03pm UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/2 "2021-12-09T20:03:55Z")

</div>

Please check this

> [@Elasticserach 7.2 Unable to create cluster with error "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory"](https://discuss.elastic.co/t/elasticserach-7-2-unable-to-create-cluster-with-error-unable-to-lock-jvm-memory-error-12-reason-cannot-allocate-memory/252050/4):
>
> The problem is that you have set bootstrap.memory\_lock: true but your system does not support memory locking. The solution is therefore not to set bootstrap.memory\_lock: true. Instead you should [disable swap](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html).

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [December 10, 2021, 10:17am UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/3 "2021-12-10T10:17:13Z")

</div>

the problem in my case is that I don't have enabled bootstrap.memory\_lock: true

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [December 10, 2021, 10:51am UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/4 "2021-12-10T10:51:49Z")

</div>

What is the heap size of your vm's and how much is the RAM ?

`Cannot allocate memory` clearly means es is not able to get enough memory .

Do you see any high gc activities? Can you share the complete log file?

Also try [disabling swap](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html) if that helps .

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [December 10, 2021, 12:23pm UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/5 "2021-12-10T12:23:54Z")

</div>

Hello,  
My issue is as follow:  
First of all I think that I have enough memory - please see an attach:

 ![free](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c6fa5db0a29890280203bf1bb208d60527fcb0c5.jpeg)  
first of all I'm using below configuration from jvm.options:

```auto
## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms12g4g
## -Xms12gx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms12g represents the initial size of total heap space
# Xms12gx represents the maximum size of total heap space

-Xms20g
-Xmx20g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
#-XX:+UseConcMarkSweepGC
#-XX:CMSInitiatingOccupancyFraction=75
#-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly

#14-:-XX:+UseG1GC
#14-:-XX:G1ReservePercent=25
#14-:-XX:InitiatingHeapOccupancyPercent=30

-XX:+UseG1GC
-XX:G1ReservePercent=25
-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

```

and also I spotted that I see errors with GC Overhead:

```auto
JvmGcMonitorService", "cluster.name": "docker-cluster", "node.name": "c509678ddb86", "message": "[gc][17967] overhead, spent [472ms] collecting in the last [1s]"

```

to be honest - I don't know why Elasticsearch cannot lock JVM memory at start, and I'm not shure if my jvm.options is correct

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [December 10, 2021, 1:11pm UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/6 "2021-12-10T13:11:35Z")

</div>

is that output in Mb ? (output of `free -m` command) ?

can you run this and share the output of :

GET \_cat/nodes?v=true&h=name,node\*,heap\*  
or  
`curl -XGET "http://<ip>:9200/_cat/nodes?v=true&h=name,node*,heap*"`

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [December 10, 2021, 2:36pm UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/7 "2021-12-10T14:36:03Z")

</div>

below screenshot from heap memory, looks like ok  
 ![heap](https://us1.discourse-cdn.com/elastic/original/3X/5/4/547b3a89b2f582409a86532201b3f2caa4235b6f.jpeg)  
but I founded abother issue,  
I saw defines vm.swappiness=30 and I think this can be an issue, as I know it should have 1 or 0

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [December 10, 2021, 2:48pm UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/8 "2021-12-10T14:48:47Z")

</div>

Yes ,That's one of the option for disabling swap as mentioned earlier .

> **[Disable swapping | Elasticsearch Guide \[7.16\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration-memory.html#swappiness)**

---

<div class="post-metadata">

**Author:** ![dominbdg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominbdg/32/102457_2.png) [@dominbdg](https://discuss.elastic.co/u/dominbdg)\
**Post date:** [December 10, 2021, 3:12pm UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/9 "2021-12-10T15:12:32Z")

</div>

I have ulimit set as unlimited.  
vm.swappiness=1 (I think it is correct)  
set in Elasticsearch.yml parameter : bootstrap.memory\_lock: true  
And still have issue that cannot lock memory

Do You know what can I do more ?

I see also another issue:  
`Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536"`

I don't know how can I increase RLIMIT ?

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [December 14, 2021, 6:25am UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/10 "2021-12-14T06:25:33Z")

</div>

you can refer :

- [ulimit: difference between hard and soft limits](https://unix.stackexchange.com/questions/29577/ulimit-difference-between-hard-and-soft-limits)
- [Linux memory limits (rlimits)](http://linuxfollies.blogspot.com/2010/10/linux-memory-limits-rlimits.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2022, 6:25am UTC](https://discuss.elastic.co/t/issues-with-jvm-lock/291318/11 "2022-01-11T06:25:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.