# Three node cluster in RED after power outage; No Route to Host despite network connectivity

**URL:** <https://discuss.elastic.co/t/three-node-cluster-in-red-after-power-outage-no-route-to-host-despite-network-connectivity/18803>\
**Category:** Elasticsearch\
**Created:** [July 22, 2014, 8:28pm UTC](https://discuss.elastic.co/t/three-node-cluster-in-red-after-power-outage-no-route-to-host-despite-network-connectivity/18803 "2014-07-22T20:28:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mhughes](https://avatars.discourse-cdn.com/v4/letter/m/85e7bf/32.png) [@mhughes](https://discuss.elastic.co/u/mhughes)\
**Post date:** [July 22, 2014, 8:28pm UTC](https://discuss.elastic.co/t/three-node-cluster-in-red-after-power-outage-no-route-to-host-despite-network-connectivity/18803/1 "2014-07-22T20:28:59Z")

</div>

I have a three-node cluster that has gone red. The nodes in the cluster  
were shut down abruptly as a result of power loss.

Cluster Health is:  
{  
"cluster\_name" : "elasticsearch",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 3,  
"number\_of\_data\_nodes" : 3,  
"active\_primary\_shards" : 43,  
"active\_shards" : 86,  
"relocating\_shards" : 2,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 4  
}

The unassigned\_shards haven't changed for hours. Looking at the logs I  
have:

node1 (repeats forever):

[2014-07-22 21:23:48,221][DEBUG][action.bulk] [Midnight Man]  
observer: timeout notification from cluster service. timeout setting [1m],  
time since start [1m]

node2 (repeats):

[2014-07-22 21:24:40,333][WARN][transport.netty] [Phantazia]  
exception caught on transport layer [[id: 0x6aba1c4e]], closing connection  
java.net.NoRouteToHostException: No route to host  
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:744)

node3 (repeats):  
[2014-07-22 20:24:39,342][WARN][transport.netty] [Master Order]  
exception caught on transport layer [[id: 0xecdd8869]], closing connection  
java.net.NoRouteToHostException: No route to host  
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
at  
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:744)

Really don't understand the "No route to host". I have network  
connectivity to and from every node, tested via ping and telnet of port  
9300.

Not really sure what to do at this point besides blowing away indices. Any  
pointers would be appreciated.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/22128676-b2e3-4e30-8000-fdce62407483%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/22128676-b2e3-4e30-8000-fdce62407483%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:14am UTC](https://discuss.elastic.co/t/three-node-cluster-in-red-after-power-outage-no-route-to-host-despite-network-connectivity/18803/2 "2017-07-06T01:14:08Z")

</div>