Could someone please look into this error and provide solution.
We have 3 aws instances of type m4.2xlarge being used for elasticsearch with 1 master node and 2 shards; total of 3 indices , one of the index is holding 13 millions of data.
all of a sudden one of the instances is terminated due to EC2 health check failed and then ES cluster was not in green. with in few mins , new instance was created re-launched and added to ES cluster and turned to green.
question is why instances was terminated , was this due to memory issue ? I could see 'Unable to lock JVM Memory' on all 3 instances but only one of them got terminated. please help us understand the rootcause and resolution.
[root@IPAddress]# curl -X GET "localhost:9200/_nodes?filter_path=**.mlockall"
{"nodes":{"bosyXTN_S8iLjuFzbdOjsw":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}},"rW4OoamrTSyj7_IaNtkm5A":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}},"N3tgAJtfTtakGwwkNWimIA":{"settings":{"bootstrap":{"mlockall":"true"}},"process":{"mlockall":false}}}}
Logs
[2019-01-05 13:37:11,688][INFO ][node ] [elasticsearch1-prod] version[2.1.1], pid[18736], build[40e2c53/2015-12-15T13:05:55Z]
[2019-01-05 13:37:11,495][WARN ][bootstrap ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2019-01-05 13:37:11,495][WARN ][bootstrap ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2019-01-05 13:37:11,495][WARN ][bootstrap ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2019-01-05 13:37:11,494][WARN ][bootstrap ] This can result in part of the JVM being swapped out.
[2019-01-05 13:37:11,494][WARN ][bootstrap ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory
[2019-01-05 13:14:30,252][INFO ][cluster.service ] [elasticsearch2-prod] removed {{elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300},}, reason: zen-disco-receive(from master [{elasticsearch1-prod}{2T2eQar7RZGwd1u1zbzs9w}{10.88.83.166}{10.88.83.166:9300}])
[2019-01-05 13:14:30,245][WARN ][action.bulk ] [elasticsearch1-prod] [dns_prod][0] failed to perform indices:data/write/bulk[s] on node {elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300}
NodeDisconnectedException[[elasticsearch3-prod][':9300][indices:data/write/bulk[s][r]] disconnected]
[2019-01-05 13:14:30,245][WARN ][action.bulk ] [elasticsearch1-prod] [dns_prod][0] failed to perform indices:data/write/bulk[s] on node {elasticsearch3-prod}{hXW8lE1iS02Kd-o-p7Rdvw}{IP}{IP:9300}
NodeDisconnectedException[[elasticsearch3-prod][IP:9300][indices:data/write/bulk[s][r]] disconnected]