I have 3 node cluster configure in AWS Cloud which has ASG enable as shown in image below . Let say [ node-1, node-2, node3] are the 3 nodes and are hosted under same vpc subnet on different ip's.
When i login to one of the ip which is a master node and run 'service elasticsearch stop' command as a root user.
I can see master node was brought down the new data node is elected has master when i query in kibana dashboard , now we have only 2 nodes in the cluster . but in AWS it still shows 3 node running. Which causes the failures since the AWS is still accepting the request for failed node. How to resolve this?
I'm not very familiar but I guess that as the VM is still running, the traffic is still send there? May be you should configure "something" to make sure it checks the 9200 port and /cluster/_health URL may be?
BTW did you look at https://www.elastic.co/cloud? It does all that automatically and can run on AWS.
Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, APM, Logs UI, Infra UI, SIEM, Maps UI and what is coming next 
...
BTW i have run couple of other test i would like share the results
- I have a 3 node cluster ( 1 master , 2 data nodes , All 3 are eligible for master) and I shutdown one data node , Cluster state goes to yellow and back to Green after a while we see 2 nodes still running in which one of the node is master and other is data node.
Observations : I see small percentage of failures probably 10% for 2-3 mins. when i shutdown data node.
Is this common to have small % percentage of failures. ?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.