ElasticSearch version 2.1.1 -Unable to lock JVM Memory

Could someone please look into this error and provide solution.

We have 3 aws instances of type m4.2xlarge being used for elasticsearch with 1 master node and 2 shards; total of 3 indices , one of the index is holding 13 millions of data.

all of a sudden one of the instances is terminated due to EC2 health check failed and then ES cluster was not in green. with in few mins , new instance was created re-launched and added to ES cluster and turned to green.

question is why instances was terminated , was this due to memory issue ? I could see 'Unable to lock JVM Memory' on all 3 instances but only one of them got terminated. please help us understand the rootcause and resolution.

[root@IPAddress]# curl -X GET "localhost:9200/_nodes?filter_path=**.mlockall"

{"nodes":{"bosyXTN_S8iLjuFzbdOjsw":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}},"rW4OoamrTSyj7_IaNtkm5A":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}},"N3tgAJtfTtakGwwkNWimIA":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}}}}

Logs

[2019-01-05 13:37:11,688][INFO ][node ] [elasticsearch1-prod] version[2.1.1], pid[18736], build[40e2c53/2015-12-15T13:05:55Z]
[2019-01-05 13:37:11,495][WARN ][bootstrap ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-05 13:37:11,495][WARN ][bootstrap ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2019-01-05 13:37:11,495][WARN ][bootstrap ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-05 13:37:11,494][WARN ][bootstrap ] This can result in part of the JVM being swapped out.
[2019-01-05 13:37:11,494][WARN ][bootstrap ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory
[2019-01-05 13:14:30,252][INFO ][cluster.service ] [elasticsearch2-prod] removed {{elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300},}, reason: zen-disco-receive(from master [{elasticsearch1-prod}{2T2eQar7RZGwd1u1zbzs9w}{10.88.83.166}{10.88.83.166:9300}])
[2019-01-05 13:14:30,245][WARN ][action.bulk ] [elasticsearch1-prod] [dns_prod][0] failed to perform indices:data/write/bulk[s] on node {elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300}
NodeDisconnectedException[[elasticsearch3-prod][':9300][indices:data/write/bulk[s][r]] disconnected]
[2019-01-05 13:14:30,245][WARN ][action.bulk ] [elasticsearch1-prod] [dns_prod][0] failed to perform indices:data/write/bulk[s] on node {elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300}
NodeDisconnectedException[[elasticsearch3-prod][IP:9300][indices:data/write/bulk[s][r]] disconnected]

We have been using the ES on the aws instances with 32GB of RAM; never seen this issue in the past , not sure why its happening now; could you please share inputs why its happening now though no changes done in ES config files.

Not sure if the JVM memory caused the issue since its WARN message, please share your suggestions

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.