# Shard Allocation Problem
**URL:** https://discuss.elastic.co/t/shard-allocation-problem/22948
**Category:** Elasticsearch
**Created:** [March 27, 2015, 5:27pm UTC](https://discuss.elastic.co/t/shard-allocation-problem/22948 "2015-03-27T17:27:24Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)
#### Post date: [March 27, 2015, 5:27pm UTC](https://discuss.elastic.co/t/shard-allocation-problem/22948/1 "2015-03-27T17:27:24Z")
Hi guys,
Our cluster has 7 data nodes. Last week one of them was turned off. We
restarted the node and then the cluster started rebalancing. The problem is
it nerver finishes rebalancing.
Node logs shows an update coming from master node during the rebalancing,
seconds later the rebalancing is canceled and then it starts again.
I have created #10281
[https://github.com/elastic/elasticsearch/issues/10281](https://github.com/elastic/elasticsearch/issues/10281) on GitHub, but I am
including the problem here too. Maybe somebody ran into this problem and
already fixed it.
The logs were collected in different moments but they represent the same
behaviour.
The log follows:
Data Node:
[2015-03-24 15:26:05,244][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] creating shard
[2015-03-24 15:26:05,244][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [0]
[2015-03-24 15:26:05,274][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][0] Using [keep\_only\_last] deletion policy
[2015-03-24 15:26:05,275][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][0] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
[2015-03-24 15:26:05,276][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][0] using [concurrent] merge scheduler with max\_thread\_count[3]
[2015-03-24 15:26:05,278][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]
[2015-03-24 15:26:05,278][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][0] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
[2015-03-24 15:26:05,280][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]-\>[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
[2015-03-24 15:26:05,281][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] creating shard
[2015-03-24 15:26:05,281][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [2]
[2015-03-24 15:26:05,358][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][2] Using [keep\_only\_last] deletion policy
[2015-03-24 15:26:05,361][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][2] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
[2015-03-24 15:26:05,363][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][2] using [concurrent] merge scheduler with max\_thread\_count[3]
[2015-03-24 15:26:05,371][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]
[2015-03-24 15:26:05,373][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][2] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
[2015-03-24 15:26:05,378][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]-\>[RECOVERING], reason [from [PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][es4.mydomain][inet[/172.31.234.167:9300]]{master=false}]
[2015-03-24 15:26:05,380][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116348)
[2015-03-24 15:26:05,682][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116349
[2015-03-24 15:26:05,684][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
[2015-03-24 15:26:05,685][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
[2015-03-24 15:26:05,685][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116349], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
[2015-03-24 15:26:05,686][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116349
[2015-03-24 15:26:05,687][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] removing shard (different instance of it allocated on this node, current [[my\_index][0], node[OI3VgsmPSBekiVF7WhSm4Q], relocating [gBhKyL1gQ1WnHLUFvR\_BPQ], [R], s[INITIALIZING]], global [[my\_index][0], node[OI3VgsmPSBekiVF7WhSm4Q], [R], s[INITIALIZING]])
[2015-03-24 15:26:05,807][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [RECOVERING]-\>[CLOSED], reason [removing shard (different instance of it allocated on this node)]
[2015-03-24 15:26:05,808][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] creating shard
[2015-03-24 15:26:05,808][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [0]
[2015-03-24 15:26:05,856][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][0] Using [keep\_only\_last] deletion policy
[2015-03-24 15:26:05,857][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][0] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
[2015-03-24 15:26:05,858][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][0] using [concurrent] merge scheduler with max\_thread\_count[3]
[2015-03-24 15:26:05,861][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]
[2015-03-24 15:26:05,862][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][0] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
[2015-03-24 15:26:05,865][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]-\>[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
[2015-03-24 15:26:05,868][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116349)
[2015-03-24 15:26:06,514][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116350
[2015-03-24 15:26:06,514][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
[2015-03-24 15:26:06,515][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
[2015-03-24 15:26:06,515][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116350], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
[2015-03-24 15:26:06,516][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116350
[2015-03-24 15:26:06,518][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] removing shard (different instance of it allocated on this node, current [[my\_index][2], node[OI3VgsmPSBekiVF7WhSm4Q], relocating [Fto2cgJ-RB-IjV3EfLm1Sw], [R], s[INITIALIZING]], global [[my\_index][2], node[OI3VgsmPSBekiVF7WhSm4Q], [R], s[INITIALIZING]])
[2015-03-24 15:26:06,622][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [RECOVERING]-\>[CLOSED], reason [removing shard (different instance of it allocated on this node)]
[2015-03-24 15:26:06,623][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] creating shard
[2015-03-24 15:26:06,623][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [2]
[2015-03-24 15:26:06,687][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][2] Using [keep\_only\_last] deletion policy
[2015-03-24 15:26:06,688][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][2] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
[2015-03-24 15:26:06,690][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][2] using [concurrent] merge scheduler with max\_thread\_count[3]
[2015-03-24 15:26:06,692][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]
[2015-03-24 15:26:06,692][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][2] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
[2015-03-24 15:26:06,695][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]-\>[RECOVERING], reason [from [PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][es4.mydomain][inet[/172.31.234.167:9300]]{master=false}]
[2015-03-24 15:26:06,697][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116350)
[2015-03-24 15:26:08,215][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116351
[2015-03-24 15:26:08,215][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
[2015-03-24 15:26:08,216][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
[2015-03-24 15:26:08,217][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116351], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
[2015-03-24 15:26:08,218][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116351
[2015-03-24 15:26:08,224][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] removing shard (not allocated)
[2015-03-24 15:26:08,297][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [RECOVERING]-\>[CLOSED], reason [removing shard (not allocated)]
[2015-03-24 15:26:08,304][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116351)
[2015-03-24 15:26:08,812][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116352
[2015-03-24 15:26:08,813][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
[2015-03-24 15:26:08,813][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
[2015-03-24 15:26:08,814][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116352], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
[2015-03-24 15:26:08,814][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116352
[2015-03-24 15:26:08,816][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] removing shard (not allocated)
[2015-03-24 15:26:08,820][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [RECOVERING]-\>[CLOSED], reason [removing shard (not allocated)]
[2015-03-24 15:26:08,821][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index] cleaning index (no shards allocated)
[2015-03-24 15:26:08,821][DEBUG][index.cache.filter.weighted] [THE\_CHOSEN\_ONE] [my\_index] full cache clear, reason [close]
[2015-03-24 15:26:08,827][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116352)
Master node:
[2015-03-26 15:23:08,468][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,506][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,516][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,523][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,534][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,545][DEBUG][cluster.service] [MR3] set local cluster state to version 142212
[2015-03-26 15:23:08,545][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
[2015-03-26 15:23:08,545][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
[2015-03-26 15:23:08,547][DEBUG][cluster.service] [MR3] processing [shard-started ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], [R], s[INITIALIZING]), reason [after recovery (replica) from node [[KOZANI][\_dLxnGwlTt6P4i7PqzNEUQ][pves1-6.popvono][inet[/172.31.234.187:9300]]{master=false}]]]: done applying updated cluster\_state (version: 142212)
[2015-03-26 15:23:08,547][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,549][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,549][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,564][DEBUG][cluster.service] [MR3] cluster state updated, version [142213], source [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]
[2015-03-26 15:23:08,564][DEBUG][cluster.service] [MR3] publishing cluster state version 142213
[2015-03-26 15:23:08,601][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,619][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,668][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,724][DEBUG][cluster.service] [MR3] set local cluster state to version 142213
[2015-03-26 15:23:08,725][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
[2015-03-26 15:23:08,725][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
[2015-03-26 15:23:08,725][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: done applying updated cluster\_state (version: 142213)
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
[2015-03-26 15:23:08,739][DEBUG][gateway.local] [MR3] [my\_index][1]: allocating [[my\_index][1], node[null], [R], s[UNASSIGNED]] to [[PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][pves1-4.popvono][inet[/172.31.234.167:9300]]{master=false}] in order to reuse its unallocated persistent store with total\_size [7gb]
[2015-03-26 15:23:08,746][DEBUG][cluster.service] [MR3] cluster state updated, version [142214], source [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]
[2015-03-26 15:23:08,746][DEBUG][cluster.service] [MR3] publishing cluster state version 142214
[2015-03-26 15:23:08,775][DEBUG][cluster.service] [MR3] set local cluster state to version 142214
[2015-03-26 15:23:08,776][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
[2015-03-26 15:23:08,776][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: done applying updated cluster\_state (version: 142214)
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
[2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/85481982-8abe-4cf5-8ccd-655f08d38855%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85481982-8abe-4cf5-8ccd-655f08d38855%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 28, 2015, 5:47am UTC](https://discuss.elastic.co/t/shard-allocation-problem/22948/2 "2015-03-28T05:47:15Z")
Please don't put such large logs directly into emails like this, use
gist/pastebin/etc 🙂
If you can do that someone might be able to read things a little easier and
provide assistance.
On 28 March 2015 at 04:27, Marcelo Paes Rech [marcelopaesrech@gmail.com](mailto:marcelopaesrech@gmail.com)
wrote:
> Hi guys,
>
> Our cluster has 7 data nodes. Last week one of them was turned off. We
> restarted the node and then the cluster started rebalancing. The problem is
> it nerver finishes rebalancing.
>
> Node logs shows an update coming from master node during the rebalancing,
> seconds later the rebalancing is canceled and then it starts again.
>
> I have created #10281
> [https://github.com/elastic/elasticsearch/issues/10281](https://github.com/elastic/elasticsearch/issues/10281) on GitHub, but I
> am including the problem here too. Maybe somebody ran into this problem and
> already fixed it.
>
> The logs were collected in different moments but they represent the same
> behaviour.
>
> The log follows:
>
> Data Node:
>
> [2015-03-24 15:26:05,244][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] creating shard
> [2015-03-24 15:26:05,244][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [0]
> [2015-03-24 15:26:05,274][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][0] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:05,275][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][0] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:05,276][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][0] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:05,278][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]
> [2015-03-24 15:26:05,278][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][0] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:05,280][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]-\>[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
> [2015-03-24 15:26:05,281][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] creating shard
> [2015-03-24 15:26:05,281][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [2]
> [2015-03-24 15:26:05,358][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][2] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:05,361][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][2] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:05,363][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][2] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:05,371][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]
> [2015-03-24 15:26:05,373][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][2] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:05,378][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]-\>[RECOVERING], reason [from [PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][es4.mydomain][inet[/172.31.234.167:9300]]{master=false}]
> [2015-03-24 15:26:05,380][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116348)
> [2015-03-24 15:26:05,682][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116349
> [2015-03-24 15:26:05,684][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:05,685][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:05,685][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116349], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:05,686][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116349
> [2015-03-24 15:26:05,687][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] removing shard (different instance of it allocated on this node, current [[my\_index][0], node[OI3VgsmPSBekiVF7WhSm4Q], relocating [gBhKyL1gQ1WnHLUFvR\_BPQ], [R], s[INITIALIZING]], global [[my\_index][0], node[OI3VgsmPSBekiVF7WhSm4Q], [R], s[INITIALIZING]])
> [2015-03-24 15:26:05,807][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [RECOVERING]-\>[CLOSED], reason [removing shard (different instance of it allocated on this node)]
> [2015-03-24 15:26:05,808][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] creating shard
> [2015-03-24 15:26:05,808][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [0]
> [2015-03-24 15:26:05,856][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][0] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:05,857][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][0] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:05,858][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][0] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:05,861][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]
> [2015-03-24 15:26:05,862][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][0] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:05,865][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]-\>[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
> [2015-03-24 15:26:05,868][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116349)
> [2015-03-24 15:26:06,514][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116350
> [2015-03-24 15:26:06,514][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:06,515][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:06,515][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116350], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:06,516][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116350
> [2015-03-24 15:26:06,518][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] removing shard (different instance of it allocated on this node, current [[my\_index][2], node[OI3VgsmPSBekiVF7WhSm4Q], relocating [Fto2cgJ-RB-IjV3EfLm1Sw], [R], s[INITIALIZING]], global [[my\_index][2], node[OI3VgsmPSBekiVF7WhSm4Q], [R], s[INITIALIZING]])
> [2015-03-24 15:26:06,622][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [RECOVERING]-\>[CLOSED], reason [removing shard (different instance of it allocated on this node)]
> [2015-03-24 15:26:06,623][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] creating shard
> [2015-03-24 15:26:06,623][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [2]
> [2015-03-24 15:26:06,687][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][2] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:06,688][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][2] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:06,690][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][2] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:06,692][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]
> [2015-03-24 15:26:06,692][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][2] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:06,695][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]-\>[RECOVERING], reason [from [PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][es4.mydomain][inet[/172.31.234.167:9300]]{master=false}]
> [2015-03-24 15:26:06,697][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116350)
> [2015-03-24 15:26:08,215][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116351
> [2015-03-24 15:26:08,215][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:08,216][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:08,217][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116351], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:08,218][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116351
> [2015-03-24 15:26:08,224][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] removing shard (not allocated)
> [2015-03-24 15:26:08,297][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [RECOVERING]-\>[CLOSED], reason [removing shard (not allocated)]
> [2015-03-24 15:26:08,304][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116351)
> [2015-03-24 15:26:08,812][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116352
> [2015-03-24 15:26:08,813][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:08,813][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:08,814][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116352], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:08,814][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116352
> [2015-03-24 15:26:08,816][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] removing shard (not allocated)
> [2015-03-24 15:26:08,820][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [RECOVERING]-\>[CLOSED], reason [removing shard (not allocated)]
> [2015-03-24 15:26:08,821][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index] cleaning index (no shards allocated)
> [2015-03-24 15:26:08,821][DEBUG][index.cache.filter.weighted] [THE\_CHOSEN\_ONE] [my\_index] full cache clear, reason [close]
> [2015-03-24 15:26:08,827][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116352)
>
> Master node:
>
> [2015-03-26 15:23:08,468][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,506][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,516][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,523][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,534][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,545][DEBUG][cluster.service] [MR3] set local cluster state to version 142212
> [2015-03-26 15:23:08,545][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
> [2015-03-26 15:23:08,545][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
> [2015-03-26 15:23:08,547][DEBUG][cluster.service] [MR3] processing [shard-started ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], [R], s[INITIALIZING]), reason [after recovery (replica) from node [[KOZANI][\_dLxnGwlTt6P4i7PqzNEUQ][pves1-6.popvono][inet[/172.31.234.187:9300]]{master=false}]]]: done applying updated cluster\_state (version: 142212)
> [2015-03-26 15:23:08,547][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,549][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,549][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,564][DEBUG][cluster.service] [MR3] cluster state updated, version [142213], source [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]
> [2015-03-26 15:23:08,564][DEBUG][cluster.service] [MR3] publishing cluster state version 142213
> [2015-03-26 15:23:08,601][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,619][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,668][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,724][DEBUG][cluster.service] [MR3] set local cluster state to version 142213
> [2015-03-26 15:23:08,725][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
> [2015-03-26 15:23:08,725][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
> [2015-03-26 15:23:08,725][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: done applying updated cluster\_state (version: 142213)
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,728][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,729][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,730][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,739][DEBUG][gateway.local] [MR3] [my\_index][1]: allocating [[my\_index][1], node[null], [R], s[UNASSIGNED]] to [[PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][pves1-4.popvono][inet[/172.31.234.167:9300]]{master=false}] in order to reuse its unallocated persistent store with total\_size [7gb]
> [2015-03-26 15:23:08,746][DEBUG][cluster.service] [MR3] cluster state updated, version [142214], source [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]
> [2015-03-26 15:23:08,746][DEBUG][cluster.service] [MR3] publishing cluster state version 142214
> [2015-03-26 15:23:08,775][DEBUG][cluster.service] [MR3] set local cluster state to version 142214
> [2015-03-26 15:23:08,776][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
> [2015-03-26 15:23:08,776][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: done applying updated cluster\_state (version: 142214)
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,777][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING]), reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/85481982-8abe-4cf5-8ccd-655f08d38855%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85481982-8abe-4cf5-8ccd-655f08d38855%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/85481982-8abe-4cf5-8ccd-655f08d38855%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85481982-8abe-4cf5-8ccd-655f08d38855%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X--M92omPYtWvNpXs987zu6ni80p%3DiMRg8M4zbRGt5Luw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X--M92omPYtWvNpXs987zu6ni80p%3DiMRg8M4zbRGt5Luw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)
#### Post date: [March 28, 2015, 7:27pm UTC](https://discuss.elastic.co/t/shard-allocation-problem/22948/3 "2015-03-28T19:27:24Z")
I will create a new post for that.
Em sábado, 28 de março de 2015 02:47:44 UTC-3, Mark Walkom escreveu:
> Please don't put such large logs directly into emails like this, use
> gist/pastebin/etc 🙂
>
> If you can do that someone might be able to read things a little easier
> and provide assistance.
>
> On 28 March 2015 at 04:27, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com)
> \\> wrote:
>
> Hi guys,
>
> Our cluster has 7 data nodes. Last week one of them was turned off. We
> restarted the node and then the cluster started rebalancing. The problem is
> it nerver finishes rebalancing.
>
> Node logs shows an update coming from master node during the rebalancing,
> seconds later the rebalancing is canceled and then it starts again.
>
> I have created #10281
> [https://github.com/elastic/elasticsearch/issues/10281](https://github.com/elastic/elasticsearch/issues/10281) on GitHub, but I
> am including the problem here too. Maybe somebody ran into this problem and
> already fixed it.
>
> The logs were collected in different moments but they represent the same
> behaviour.
>
> The log follows:
>
> Data Node:
>
> [2015-03-24 15:26:05,244][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] creating shard
> [2015-03-24 15:26:05,244][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [0]
> [2015-03-24 15:26:05,274][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][0] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:05,275][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][0] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:05,276][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][0] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:05,278][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]
> [2015-03-24 15:26:05,278][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][0] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:05,280][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]-\>[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
> [2015-03-24 15:26:05,281][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] creating shard
> [2015-03-24 15:26:05,281][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [2]
> [2015-03-24 15:26:05,358][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][2] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:05,361][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][2] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:05,363][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][2] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:05,371][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]
> [2015-03-24 15:26:05,373][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][2] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:05,378][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]-\>[RECOVERING], reason [from [PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][es4.mydomain][inet[/172.31.234.167:9300]]{master=false}]
> [2015-03-24 15:26:05,380][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116348)
> [2015-03-24 15:26:05,682][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116349
> [2015-03-24 15:26:05,684][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:05,685][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:05,685][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116349], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:05,686][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116349
> [2015-03-24 15:26:05,687][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] removing shard (different instance of it allocated on this node, current [[my\_index][0], node[OI3VgsmPSBekiVF7WhSm4Q], relocating [gBhKyL1gQ1WnHLUFvR\_BPQ], [R], s[INITIALIZING]], global [[my\_index][0], node[OI3VgsmPSBekiVF7WhSm4Q], [R], s[INITIALIZING]])
> [2015-03-24 15:26:05,807][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [RECOVERING]-\>[CLOSED], reason [removing shard (different instance of it allocated on this node)]
> [2015-03-24 15:26:05,808][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] creating shard
> [2015-03-24 15:26:05,808][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [0]
> [2015-03-24 15:26:05,856][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][0] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:05,857][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][0] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:05,858][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][0] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:05,861][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]
> [2015-03-24 15:26:05,862][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][0] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:05,865][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [CREATED]-\>[RECOVERING], reason [from [VEROIA][Fto2cgJ-RB-IjV3EfLm1Sw][es2.mydomain][inet[/172.31.234.165:9300]]{master=false}]
> [2015-03-24 15:26:05,868][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116349)
> [2015-03-24 15:26:06,514][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116350
> [2015-03-24 15:26:06,514][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:06,515][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:06,515][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116350], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:06,516][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116350
> [2015-03-24 15:26:06,518][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] removing shard (different instance of it allocated on this node, current [[my\_index][2], node[OI3VgsmPSBekiVF7WhSm4Q], relocating [Fto2cgJ-RB-IjV3EfLm1Sw], [R], s[INITIALIZING]], global [[my\_index][2], node[OI3VgsmPSBekiVF7WhSm4Q], [R], s[INITIALIZING]])
> [2015-03-24 15:26:06,622][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [RECOVERING]-\>[CLOSED], reason [removing shard (different instance of it allocated on this node)]
> [2015-03-24 15:26:06,623][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] creating shard
> [2015-03-24 15:26:06,623][DEBUG][index.service] [THE\_CHOSEN\_ONE] [my\_index] creating shard\_id [2]
> [2015-03-24 15:26:06,687][DEBUG][index.deletionpolicy] [THE\_CHOSEN\_ONE] [my\_index][2] Using [keep\_only\_last] deletion policy
> [2015-03-24 15:26:06,688][DEBUG][index.merge.policy] [THE\_CHOSEN\_ONE] [my\_index][2] using [tiered] merge policy with expunge\_deletes\_allowed[10.0], floor\_segment[2mb], max\_merge\_at\_once[10], max\_merge\_at\_once\_explicit[30], max\_merged\_segment[5gb], segments\_per\_tier[10.0], reclaim\_deletes\_weight[2.0]
> [2015-03-24 15:26:06,690][DEBUG][index.merge.scheduler] [THE\_CHOSEN\_ONE] [my\_index][2] using [concurrent] merge scheduler with max\_thread\_count[3]
> [2015-03-24 15:26:06,692][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]
> [2015-03-24 15:26:06,692][DEBUG][index.translog] [THE\_CHOSEN\_ONE] [my\_index][2] interval [5s], flush\_threshold\_ops [5000], flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2015-03-24 15:26:06,695][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [CREATED]-\>[RECOVERING], reason [from [PERAIA][bOUh4S-fSmW8vcD8Gw0sQw][es4.mydomain][inet[/172.31.234.167:9300]]{master=false}]
> [2015-03-24 15:26:06,697][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116350)
> [2015-03-24 15:26:08,215][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116351
> [2015-03-24 15:26:08,215][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:08,216][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:08,217][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116351], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:08,218][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116351
> [2015-03-24 15:26:08,224][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][0] removing shard (not allocated)
> [2015-03-24 15:26:08,297][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][0] state: [RECOVERING]-\>[CLOSED], reason [removing shard (not allocated)]
> [2015-03-24 15:26:08,304][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116351)
> [2015-03-24 15:26:08,812][DEBUG][discovery.zen.publish] [THE\_CHOSEN\_ONE] received cluster state version 116352
> [2015-03-24 15:26:08,813][DEBUG][discovery.zen] [THE\_CHOSEN\_ONE] received cluster state from [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}] which is also master but with cluster name [Cluster [gvtmusic]]
> [2015-03-24 15:26:08,813][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: execute
> [2015-03-24 15:26:08,814][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] cluster state updated, version [116352], source [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]
> [2015-03-24 15:26:08,814][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] set local cluster state to version 116352
> [2015-03-24 15:26:08,816][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index][2] removing shard (not allocated)
> [2015-03-24 15:26:08,820][DEBUG][index.shard.service] [THE\_CHOSEN\_ONE] [my\_index][2] state: [RECOVERING]-\>[CLOSED], reason [removing shard (not allocated)]
> [2015-03-24 15:26:08,821][DEBUG][indices.cluster] [THE\_CHOSEN\_ONE] [my\_index] cleaning index (no shards allocated)
> [2015-03-24 15:26:08,821][DEBUG][index.cache.filter.weighted] [THE\_CHOSEN\_ONE] [my\_index] full cache clear, reason [close]
> [2015-03-24 15:26:08,827][DEBUG][cluster.service] [THE\_CHOSEN\_ONE] processing [zen-disco-receive(from master [[MR3][4LssjGYPTX6EL7Kv-ylgMQ][master3.mydomain][inet[/172.31.234.181:9300]]{data=false, master=true}])]: done applying updated cluster\_state (version: 116352)
>
> Master node:
>
> [2015-03-26 15:23:08,468][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,506][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,516][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,523][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,534][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,545][DEBUG][cluster.service] [MR3] set local cluster state to version 142212
> [2015-03-26 15:23:08,545][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
> [2015-03-26 15:23:08,545][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
> [2015-03-26 15:23:08,547][DEBUG][cluster.service] [MR3] processing [shard-started ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], [R], s[INITIALIZING]), reason [after recovery (replica) from node [[KOZANI][\_dLxnGwlTt6P4i7PqzNEUQ][pves1-6.popvono][inet[/172.31.234.187:9300]]{master=false}]]]: done applying updated cluster\_state (version: 142212)
> [2015-03-26 15:23:08,547][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,547][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,548][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,549][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,549][DEBUG][cluster.action.shard] [MR3] [my\_index][1] will apply shard failed [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,564][DEBUG][cluster.service] [MR3] cluster state updated, version [142213], source [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]
> [2015-03-26 15:23:08,564][DEBUG][cluster.service] [MR3] publishing cluster state version 142213
> [2015-03-26 15:23:08,601][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,619][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,668][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,724][DEBUG][cluster.service] [MR3] set local cluster state to version 142213
> [2015-03-26 15:23:08,725][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: execute
> [2015-03-26 15:23:08,725][DEBUG][river.cluster] [MR3] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state
> [2015-03-26 15:23:08,725][WARN][cluster.action.shard] [MR3] [my\_index][1] received shard failed for [my\_index][1], node[4UkjJGtNQdCvJH3OuXEyUQ], [R], s[INITIALIZING], indexUUID [evIJHzdjQP6x0vr3Tve2-w], reason [Failed to perform [index] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: done applying updated cluster\_state (version: 142213)
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: no change in cluster\_state
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize exception response from stream]; nested: StreamCorruptedException[unexpected end of block data]; ]]]: execute
> [2015-03-26 15:23:08,727][DEBUG][cluster.service] [MR3] processing [shard-failed ([my\_index][1], node[bOUh4S-fSmW8vcD8Gw0sQw], relocating [4UkjJGtNQdCvJH3OuXEyUQ], [R], s[RELOCATING]), reason [Failed to perform [indices/index/b\_shard/delete] on replica, message [RemoteTransportException[Failed to deserialize exception response from stream]; nested: TransportSerializationException[Failed to deserialize excep
>
> ...
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/956fd660-0a6f-441a-aecd-736537898b49%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/956fd660-0a6f-441a-aecd-736537898b49%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:23am UTC](https://discuss.elastic.co/t/shard-allocation-problem/22948/4 "2017-07-06T00:23:15Z")