# ElasticSearch version 2.1.1 -Unable to lock JVM Memory **URL:** \ **Category:** Elasticsearch\ **Created:** [January 7, 2019, 9:43pm UTC](https://discuss.elastic.co/t/elasticsearch-version-2-1-1-unable-to-lock-jvm-memory/163274 "2019-01-07T21:43:08Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Neni](https://avatars.discourse-cdn.com/v4/letter/n/a6a055/32.png) [@Neni](https://discuss.elastic.co/u/Neni)\ **Post date:** [January 7, 2019, 9:43pm UTC](https://discuss.elastic.co/t/elasticsearch-version-2-1-1-unable-to-lock-jvm-memory/163274/1 "2019-01-07T21:43:09Z")
Could someone please look into this error and provide solution. We have 3 aws instances of type m4.2xlarge being used for elasticsearch with 1 master node and 2 shards; total of 3 indices , one of the index is holding 13 millions of data. all of a sudden one of the instances is terminated due to EC2 health check failed and then ES cluster was not in green. with in few mins , new instance was created re-launched and added to ES cluster and turned to green. question is why instances was terminated , was this due to memory issue ? I could see 'Unable to lock JVM Memory' on all 3 instances but only one of them got terminated. please help us understand the rootcause and resolution. [root@IPAddress]# curl -X GET "localhost:9200/\_nodes?filter\_path=\*\*.mlockall" {"nodes":{"bosyXTN\_S8iLjuFzbdOjsw":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}},"rW4OoamrTSyj7\_IaNtkm5A":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}},"N3tgAJtfTtakGwwkNWimIA":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}}}} **Logs** [2019-01-05 13:37:11,688][INFO][node] [elasticsearch1-prod] version[2.1.1], pid[18736], build[40e2c53/2015-12-15T13:05:55Z] [2019-01-05 13:37:11,495][WARN][bootstrap] If you are logged in interactively, you will have to re-login for the new limits to take effect. [2019-01-05 13:37:11,495][WARN][bootstrap] These can be adjusted by modifying /etc/security/limits.conf, for example: # allow user 'elasticsearch' mlockall elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited [2019-01-05 13:37:11,495][WARN][bootstrap] Increase RLIMIT\_MEMLOCK, soft limit: 65536, hard limit: 65536 [2019-01-05 13:37:11,494][WARN][bootstrap] This can result in part of the JVM being swapped out. [2019-01-05 13:37:11,494][WARN][bootstrap] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory [2019-01-05 13:14:30,252][INFO][cluster.service] [elasticsearch2-prod] removed {{elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300},}, reason: zen-disco-receive(from master [{elasticsearch1-prod}{2T2eQar7RZGwd1u1zbzs9w}{10.88.83.166}{10.88.83.166:9300}]) [2019-01-05 13:14:30,245][WARN][action.bulk] [elasticsearch1-prod] [dns\_prod][0] failed to perform indices:data/write/bulk[s] on node {elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300} NodeDisconnectedException[[elasticsearch3-prod][':9300][indices:data/write/bulk[s][r]] disconnected] [2019-01-05 13:14:30,245][WARN][action.bulk] [elasticsearch1-prod] [dns\_prod][0] failed to perform indices:data/write/bulk[s] on node {elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300} NodeDisconnectedException[[elasticsearch3-prod][IP:9300][indices:data/write/bulk[s][r]] disconnected] ---
**Author:** ![Neni](https://avatars.discourse-cdn.com/v4/letter/n/a6a055/32.png) [@Neni](https://discuss.elastic.co/u/Neni)\ **Post date:** [January 7, 2019, 10:48pm UTC](https://discuss.elastic.co/t/elasticsearch-version-2-1-1-unable-to-lock-jvm-memory/163274/2 "2019-01-07T22:48:32Z")
We have been using the ES on the aws instances with 32GB of RAM; never seen this issue in the past , not sure why its happening now; could you please share inputs why its happening now though no changes done in ES config files. ---
**Author:** ![arghya.b](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@arghya.b](https://discuss.elastic.co/u/arghya.b)\ **Post date:** [January 11, 2019, 6:43pm UTC](https://discuss.elastic.co/t/elasticsearch-version-2-1-1-unable-to-lock-jvm-memory/163274/3 "2019-01-11T18:43:10Z")
Not sure if the JVM memory caused the issue since its WARN message, please share your suggestions ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 8, 2019, 6:47pm UTC](https://discuss.elastic.co/t/elasticsearch-version-2-1-1-unable-to-lock-jvm-memory/163274/4 "2019-02-08T18:47:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.