# Elasticsearch 1.4 node loss discovery **URL:** \ **Category:** Elasticsearch\ **Created:** [March 24, 2016, 3:52pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-node-loss-discovery/45366 "2016-03-24T15:52:38Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![bevans88](https://avatars.discourse-cdn.com/v4/letter/b/a698b9/32.png) [@bevans88](https://discuss.elastic.co/u/bevans88)\ **Post date:** [March 24, 2016, 3:52pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-node-loss-discovery/45366/1 "2016-03-24T15:52:38Z")
I'm currently testing Elasticsearch resilience (version 1.4.5) when a routing allocation zone is lost. These zones are AWS availability zones. The cluster configuration is 6 nodes, all master eligible, with 3 zones, 2 in each, so; es01- Zone A - Master Node es02- Zone B es03- Zone C es04- Zone A es05- Zone B es06- Zone C The process is simple: - Stop issued to es03 and es06 via AWS console. The cluster then takes around 15 minutes to return to an operable state; - For around 15 minutes the cluster health reports as having 6 nodes (this should be 4) and a green health. - After this it changes to 4, with the logs showing that es03 and es06 have been removed. - For nearly all of this time the logs show: > failed to list shard stores on node [gn0RQW3kQQCKQl6x1aL2pw] > org.elasticsearch.action.FailedNodeException: Failed node [gn0RQW3kQQCKQl6x1aL2pw] > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onFailure(TransportNodesOperationAction.java:206) > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$1000(TransportNodesOperationAction.java:97) > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4.handleException(TransportNodesOperationAction.java:178) > at org.elasticsearch.transport.TransportService$3.run(TransportService.java:217) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.elasticsearch.transport.SendRequestTransportException: [pp-pod2-es03-apollo][inet[/10.244.43.10:9300]][internal:cluster/nodes/indices/shard/store[n]] > at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:213) > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.start(TransportNodesOperationAction.java:165) > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$300(TransportNodesOperationAction.java:97) > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:70) > at org.elasticsearch.action.support.nodes.TransportNodesOperationAction.doExecute(TransportNodesOperationAction.java:43) > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) > at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:55) > at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.list(TransportNodesListShardStoreMetaData.java:79) > at org.elasticsearch.gateway.local.LocalGatewayAllocator.buildShardStores(LocalGatewayAllocator.java:454) > at org.elasticsearch.gateway.local.LocalGatewayAllocator.allocateUnassigned(LocalGatewayAllocator.java:292) > at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocators.allocateUnassigned(ShardsAllocators.java:74) > at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:217) > at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:160) > at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:146) > at org.elasticsearch.discovery.zen.ZenDiscovery$5.execute(ZenDiscovery.java:538) > at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:347) > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:184) > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:154) > ... 3 more > Caused by: org.elasticsearch.transport.NodeNotConnectedException: [pp-pod2-es03-apollo][inet[/10.244.43.10:9300]] Node not connected > at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:946) > at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:640) > at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:199) > ... 20 more The cluster configuration is: > cluster.name: testcluster > cluster.routing.allocation.awareness.atrributes: zone > discovery.zen.minimum\_master\_nodes: 4 > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: ["es01", "es02", "es03", "es04", "es05", "es06"] > jmx.create\_connector: true > node.max\_local\_storage\_nodes: 1 > node.name: es01 > node.zone: Zone A > path.data: /tmp/es01 > threadpool.bulk.queue\_size: 100 > threadpool.bulk.size: 30 > threadpool.bulk.type: fixed > action.auto\_create\_index: false Is anyone able to provide any advice on why it takes so long for the cluster to detect that es03 and es06 have been lost? Thanks, Brent ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 25, 2016, 8:28pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-node-loss-discovery/45366/2 "2016-03-25T20:28:30Z")
I'd start by upgrading, there are known issues with older versions that can cause data corruption and other general resiliency issues. See [https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html](https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html) for more. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:05pm UTC](https://discuss.elastic.co/t/elasticsearch-1-4-node-loss-discovery/45366/3 "2017-07-05T23:05:04Z")