# Network interruption, some nodes not recovering
**URL:** https://discuss.elastic.co/t/network-interruption-some-nodes-not-recovering/6639
**Category:** Elasticsearch
**Created:** [February 8, 2012, 5:00pm UTC](https://discuss.elastic.co/t/network-interruption-some-nodes-not-recovering/6639 "2012-02-08T17:00:07Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@Grant](https://discuss.elastic.co/u/Grant)
#### Post date: [February 8, 2012, 5:00pm UTC](https://discuss.elastic.co/t/network-interruption-some-nodes-not-recovering/6639/1 "2012-02-08T17:00:07Z")
So I've had this happen a couple times, where network issues lead to
some number of nodes in our cluster not rejoining:
es-r08 =\> this node dropped to yellow and failed to rejoin when all
the other nodes had reformed a cluster and gone green:
[2012-02-08 16:11:49,757][INFO][discovery.zen] [prod-es-
r08] master\_left [[prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A][inet[/
10.182.14.95:9300]]], reason [failed to ping, tried [3] times, each
with maximum [30s] timeout]
[2012-02-08 16:11:49,766][INFO][cluster.service] [prod-es-
r08] master {new [prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw][inet[prod-es-
[r04.ihost.brewster.com/10.180.35.110:9300](http://r04.ihost.brewster.com/10.180.35.110:9300)]], previous [prod-es-r02][-
OxyqMCbSxWi9DbVcPtH2A][inet[[prod-es-r02.ihost.brewster.com/](http://prod-es-r02.ihost.brewster.com/)
10.182.14.95:9300]]}, removed {[prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A]
[inet[[prod-es-r02.ihost.brewster.com/10.182.14.95:9300](http://prod-es-r02.ihost.brewster.com/10.182.14.95:9300)]],}, reason:
zen-disco-master\_failed ([prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A][inet[/
10.182.14.95:9300]])
[2012-02-08 16:11:50,772][INFO][discovery.zen] [prod-es-
r08] master\_left [[prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw][inet[prod-es-
[r04.ihost.brewster.com/10.180.35.110:9300](http://r04.ihost.brewster.com/10.180.35.110:9300)]]], reason [no longer
master]
[2012-02-08 16:11:50,772][INFO][cluster.service] [prod-es-
r08] master {new [prod-es-r08][IO4LAR4eSX6d9tQdQ5f4YQ][inet[prod-es-
[r08.ihost.brewster.com/10.180.48.255:9300](http://r08.ihost.brewster.com/10.180.48.255:9300)]], previous [prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[[prod-es-r04.ihost.brewster.com/](http://prod-es-r04.ihost.brewster.com/)
10.180.35.110:9300]]}, removed {[prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw]
[inet[[prod-es-r04.ihost.brewster.com/10.180.35.110:9300](http://prod-es-r04.ihost.brewster.com/10.180.35.110:9300)]],}, reason:
zen-disco-master\_failed ([prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw]
[inet[[prod-es-r04.ihost.brewster.com/10.180.35.110:9300](http://prod-es-r04.ihost.brewster.com/10.180.35.110:9300)]])
[2012-02-08 16:11:52,651][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:52,805][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:52,841][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:53,318][WARN][http.netty] [prod-es-
r08] Caught exception while handling client http traffic, closing
connection [id: 0x52fa512f, /10.182.14.95:36064 =\> /
10.180.48.255:9200]
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
103)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2012-02-08 16:11:53,831][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:53,901][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:54,438][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:54,819][WARN][http.netty] [prod-es-
r08] Caught exception while handling client http traffic, closing
connection [id: 0x455a08c8, /10.182.14.95:36084 =\> /
10.180.48.255:9200]
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
103)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2012-02-08 16:11:55,278][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:11:56,319][WARN][http.netty] [prod-es-
r08] Caught exception while handling client http traffic, closing
connection [id: 0x073a321f, /10.182.14.95:36091 =\> /
10.180.48.255:9200]
java.lang.IllegalArgumentException: empty text
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:
103)
at
org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:
68)
at
org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:
81)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
198)
at
org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:
107)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:
470)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:
443)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:
80)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline
$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
783)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:
81)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:
559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
274)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:
261)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:
351)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:
282)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:
202)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:44)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2012-02-08 16:11:56,360][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:12:02,606][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:12:03,194][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:12:03,674][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:12:03,981][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
[2012-02-08 16:12:04,196][WARN][discovery.zen] [prod-es-
r08] master should not receive new cluster state from [[prod-es-r04]
[1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]]
I restart the node and it rejoins:
[2012-02-08 16:17:47,913][INFO][node] [prod-es-
r08] {0.18.7}[10530]: stopping ...
[2012-02-08 16:17:48,086][INFO][node] [prod-es-
r08] {0.18.7}[10530]: stopped
[2012-02-08 16:17:48,086][INFO][node] [prod-es-
r08] {0.18.7}[10530]: closing ...
[2012-02-08 16:17:48,141][INFO][node] [prod-es-
r08] {0.18.7}[10530]: closed
[2012-02-08 16:18:01,147][INFO][bootstrap]
max\_open\_files [65507]
[2012-02-08 16:18:01,253][WARN][common.jna] Unknown
mlockall error 0
[2012-02-08 16:18:01,263][INFO][node] [prod-es-
r08] {0.18.7}[21023]: initializing ...
[2012-02-08 16:18:01,332][INFO][plugins] [prod-es-
r08] loaded [transport-thrift, hashing-analyzer], sites [bigdesk,
transport-thrift, head]
[2012-02-08 16:18:04,667][INFO][node] [prod-es-
r08] {0.18.7}[21023]: initialized
[2012-02-08 16:18:04,667][INFO][node] [prod-es-
r08] {0.18.7}[21023]: starting ...
[2012-02-08 16:18:04,699][INFO][thrift] [prod-es-
r08] bound on port [9500]
[2012-02-08 16:18:04,785][INFO][transport] [prod-es-
r08] bound\_address {inet[/10.180.48.255:9300]}, publish\_address {inet[/
10.180.48.255:9300]}
[2012-02-08 16:18:07,965][INFO][cluster.service] [prod-es-
r08] detected\_master [prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw][inet[/
10.180.35.110:9300]], added {[prod-es-r07][V3ru\_hjBTaqgzMa7XQ\_E\_Q]
[inet[/10.180.48.216:9300]],[prod-es-r01][aGaKZ7oqT6-uTPktGffjdg]
[inet[/10.180.48.178:9300]],[prod-es-r03][f-xDmnnCS26qqECIZiR68w]
[inet[/10.182.14.97:9300]],[prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw][inet[/
10.180.35.110:9300]],[prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A][inet[/
10.182.14.95:9300]],[prod-es-r05][tRwy2ed6Tpay3ay0iinDJA][inet[/
10.180.39.14:9300]],[prod-es-r06][K9y2GoEwTRqLhN-Kcx5jjw][inet[/
10.180.46.203:9300]],}, reason: zen-disco-receive(from master [[prod-
es-r04][1uUMHJs\_T7aj\_3GNcJlZVw][inet[/10.180.35.110:9300]]])
[2012-02-08 16:18:08,045][INFO][discovery] [prod-es-
r08] brewster/XUTG4WkaRoe6KKRcI4NnkA
[2012-02-08 16:18:08,308][INFO][http] [prod-es-
r08] bound\_address {inet[/10.180.48.255:9200]}, publish\_address {inet[/
10.180.48.255:9200]}
[2012-02-08 16:18:08,310][INFO][node] [prod-es-
r08] {0.18.7}[21023]: started
es-r04:
[2012-02-08 16:11:52,589][INFO][discovery.zen] [prod-es-
r04] master\_left [[prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A][inet[prod-es-
[r02.ihost.brewster.com/10.182.14.95:9300](http://r02.ihost.brewster.com/10.182.14.95:9300)]]], reason [failed to ping,
tried [3] times, each with maximum [30s] timeout]
[2012-02-08 16:11:52,590][INFO][cluster.service] [prod-es-
r04] master {new [prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw][inet[prod-es-
[r04.ihost.brewster.com/10.180.35.110:9300](http://r04.ihost.brewster.com/10.180.35.110:9300)]], previous [prod-es-r02][-
OxyqMCbSxWi9DbVcPtH2A][inet[[prod-es-r02.ihost.brewster.com/](http://prod-es-r02.ihost.brewster.com/)
10.182.14.95:9300]]}, removed {[prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A]
[inet[[prod-es-r02.ihost.brewster.com/10.182.14.95:9300](http://prod-es-r02.ihost.brewster.com/10.182.14.95:9300)]],}, reason:
zen-disco-master\_failed ([prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A]
[inet[[prod-es-r02.ihost.brewster.com/10.182.14.95:9300](http://prod-es-r02.ihost.brewster.com/10.182.14.95:9300)]])
[2012-02-08 16:11:52,829][WARN][indices.cluster] [prod-es-
r04] [contact\_documents-5-0][0] failed to start shard
org.elasticsearch.indices.recovery.RecoveryFailedException: Index
Shard [contact\_documents-5-0][0]: Recovery failed from [prod-es-r08]
[IO4LAR4eSX6d9tQdQ5f4YQ][inet[[prod-es-r08.ihost.brewster.com/](http://prod-es-r08.ihost.brewster.com/)
10.180.48.255:9300]] into [prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw]
[inet[[prod-es-r04.ihost.brewster.com/10.180.35.110:9300](http://prod-es-r04.ihost.brewster.com/10.180.35.110:9300)]]
at
org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:
263)
at org.elasticsearch.indices.recovery.RecoveryTarget.access
$100(RecoveryTarget.java:73)
at org.elasticsearch.indices.recovery.RecoveryTarget
$2.run(RecoveryTarget.java:161)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.transport.RemoteTransportException: [prod-
es-r08][inet[/10.180.48.255:9300]][index/shard/recovery/startRecovery]
Caused by:
org.elasticsearch.transport.NotSerializableTransportException:
[org.elasticsearch.index.engine.RecoveryEngineException]
[contact\_documents-5-0][0] Phase[1] Execution failed;
[contact\_documents-5-0][0] Failed to transfer [36] files with total
size of [15.2mb]; [prod-es-r04][inet[/10.180.35.110:9300]][index/shard/
recovery/filesInfo]; [prod-es-r04][inet[/10.180.35.110:9300]] Node not
connected;
[2012-02-08 16:11:52,830][WARN][cluster.action.shard] [prod-es-
r04] sending failed shard for [contact\_documents-5-0][0],
node[1uUMHJs\_T7aj\_3GNcJlZVw], [R], s[INITIALIZING], reason [Failed to
start shard, message [RecoveryFailedException[Index Shard
[contact\_documents-5-0][0]: Recovery failed from [prod-es-r08]
[IO4LAR4eSX6d9tQdQ5f4YQ][inet[[prod-es-r08.ihost.brewster.com/](http://prod-es-r08.ihost.brewster.com/)
10.180.48.255:9300]] into [prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw]
[inet[[prod-es-r04.ihost.brewster.com/10.180.35.110:9300](http://prod-es-r04.ihost.brewster.com/10.180.35.110:9300)]]]; nested:
RemoteTransportException[[prod-es-r08][inet[/10.180.48.255:9300]]
[index/shard/recovery/startRecovery]]; nested:
NotSerializableTransportException[[org.elasticsearch.index.engine.RecoveryEngineException]
[contact\_documents-5-0][0] Phase[1] Execution failed;
[contact\_documents-5-0][0] Failed to transfer [36] files with total
size of [15.2mb]; [prod-es-r04][inet[/10.180.35.110:9300]][index/shard/
recovery/filesInfo]; [prod-es-r04][inet[/10.180.35.110:9300]] Node not
connected; ]; ]]
[2012-02-08 16:11:52,830][WARN][cluster.action.shard] [prod-es-
r04] received shard failed for [contact\_documents-5-0][0],
node[1uUMHJs\_T7aj\_3GNcJlZVw], [R], s[INITIALIZING], reason [Failed to
start shard, message [RecoveryFailedException[Index Shard
[contact\_documents-5-0][0]: Recovery failed from [prod-es-r08]
[IO4LAR4eSX6d9tQdQ5f4YQ][inet[[prod-es-r08.ihost.brewster.com/](http://prod-es-r08.ihost.brewster.com/)
10.180.48.255:9300]] into [prod-es-r04][1uUMHJs\_T7aj\_3GNcJlZVw]
[inet[[prod-es-r04.ihost.brewster.com/10.180.35.110:9300](http://prod-es-r04.ihost.brewster.com/10.180.35.110:9300)]]]; nested:
RemoteTransportException[[prod-es-r08][inet[/10.180.48.255:9300]]
[index/shard/recovery/startRecovery]]; nested:
NotSerializableTransportException[[org.elasticsearch.index.engine.RecoveryEngineException]
[contact\_documents-5-0][0] Phase[1] Execution failed;
[contact\_documents-5-0][0] Failed to transfer [36] files with total
size of [15.2mb]; [prod-es-r04][inet[/10.180.35.110:9300]][index/shard/
recovery/filesInfo]; [prod-es-r04][inet[/10.180.35.110:9300]] Node not
connected; ]; ]]
[2012-02-08 16:11:56,355][INFO][cluster.service] [prod-es-
r04] added {[prod-es-r02][-OxyqMCbSxWi9DbVcPtH2A][inet[/
10.182.14.95:9300]],}, reason: zen-disco-receive(join from node[[prod-
es-r02][-OxyqMCbSxWi9DbVcPtH2A][inet[/10.182.14.95:9300]]])
[2012-02-08 16:17:48,097][INFO][cluster.service] [prod-es-
r04] removed {[prod-es-r08][IO4LAR4eSX6d9tQdQ5f4YQ][inet[prod-es-
[r08.ihost.brewster.com/10.180.48.255:9300](http://r08.ihost.brewster.com/10.180.48.255:9300)]],}, reason: zen-disco-
node\_failed([prod-es-r08][IO4LAR4eSX6d9tQdQ5f4YQ][inet[/
10.180.48.255:9300]]), reason transport disconnected (with verified
connect)
[2012-02-08 16:18:07,860][INFO][cluster.service] [prod-es-
r04] added {[prod-es-r08][XUTG4WkaRoe6KKRcI4NnkA][inet[/
10.180.48.255:9300]],}, reason: zen-disco-receive(join from node[[prod-
es-r08][XUTG4WkaRoe6KKRcI4NnkA][inet[/10.180.48.255:9300]]])
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:40am UTC](https://discuss.elastic.co/t/network-interruption-some-nodes-not-recovering/6639/2 "2017-07-06T03:40:07Z")