# River Failover

**URL:** <https://discuss.elastic.co/t/river-failover/4165>\
**Category:** Elasticsearch\
**Created:** [March 28, 2011, 9:54am UTC](https://discuss.elastic.co/t/river-failover/4165 "2011-03-28T09:54:13Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [March 28, 2011, 9:54am UTC](https://discuss.elastic.co/t/river-failover/4165/1 "2011-03-28T09:54:13Z")

</div>

Hi,

I am just getting started with elasticsearch and I am running a  
testcluster with 5 nodes. The indexing is getting done by a river.  
After one node rebooted the river is not being started automatically  
on another node. curl -XGET still shows that te river would still be  
running  
on the dead node. Am I missing a point?

Best,  
Michel.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 28, 2011, 10:20am UTC](https://discuss.elastic.co/t/river-failover/4165/2 "2011-03-28T10:20:42Z")

</div>

The river should be restarted on another node. Maybe you can recreate and explain the states that it happens?  
On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:

> Hi,
> 
> I am just getting started with elasticsearch and I am running a  
> testcluster with 5 nodes. The indexing is getting done by a river.  
> After one node rebooted the river is not being started automatically  
> on another node. curl -XGET still shows that te river would still be  
> running  
> on the dead node. Am I missing a point?
> 
> Best,  
> Michel.

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [March 28, 2011, 11:07am UTC](https://discuss.elastic.co/t/river-failover/4165/3 "2011-03-28T11:07:35Z")

</div>

I am currently trying to recreate the issue. Could it be that it has  
something to do that, as I am using the default  
configuration without reconfiguring replication or sharding, a part of  
the data is simply not available leading the river to not  
restart itself on another node?

On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> The river should be restarted on another node. Maybe you can recreate and  
> explain the states that it happens?
> 
> On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> 
> Hi,
> 
> I am just getting started with elasticsearch and I am running a  
> testcluster with 5 nodes. The indexing is getting done by a river.  
> After one node rebooted the river is not being started automatically  
> on another node. curl -XGET still shows that te river would still be  
> running  
> on the dead node. Am I missing a point?
> 
> Best,  
> Michel.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [March 28, 2011, 11:30pm UTC](https://discuss.elastic.co/t/river-failover/4165/4 "2011-03-28T23:30:01Z")

</div>

No, defaults should work and support for river failover... . When you have more info, ping back.  
On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:

> I am currently trying to recreate the issue. Could it be that it has  
> something to do that, as I am using the default  
> configuration without reconfiguring replication or sharding, a part of  
> the data is simply not available leading the river to not  
> restart itself on another node?
> 
> On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > The river should be restarted on another node. Maybe you can recreate and  
> > explain the states that it happens?
> > 
> > On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> > 
> > Hi,
> > 
> > I am just getting started with elasticsearch and I am running a  
> > testcluster with 5 nodes. The indexing is getting done by a river.  
> > After one node rebooted the river is not being started automatically  
> > on another node. curl -XGET still shows that te river would still be  
> > running  
> > on the dead node. Am I missing a point?
> > 
> > Best,  
> > Michel.

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 12, 2011, 10:55am UTC](https://discuss.elastic.co/t/river-failover/4165/5 "2011-04-12T10:55:43Z")

</div>

The issue reappereas on the test cluster, when I do the following steps:

1. reset the database by deleting all the storage folders
2. starting elasticsearch
3. creating the river

-- now the river runs on one node, in this case 192.168.1.190

1. when I kill the this node and immediately restart it, the river is  
not being restarted, the status of the river stays like this:

curl [http://127.0.0.1:9200/\_river/dsearch\_all/\_status?pretty=true](http://127.0.0.1:9200/_river/dsearch_all/_status?pretty=true)  
{  
"\_index" : "\_river",  
"\_type" : "dsearch\_all",  
"\_id" : "\_status",  
"\_version" : 1, "\_source" :  
{"ok":true,"node":{"id":"toBTLyVKTIyvcRFHNp-RFg","name":"Marlene  
Alraune","transport\_address":"inet[/192.168.1.190:9300]"}}  
}

additionally if the riverthread on the node is terminated by an  
exception, the river is not restarted automatically.

On Tue, Mar 29, 2011 at 1:30 AM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> No, defaults should work and support for river failover... . When you have  
> more info, ping back.
> 
> On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:
> 
> I am currently trying to recreate the issue. Could it be that it has  
> something to do that, as I am using the default  
> configuration without reconfiguring replication or sharding, a part of  
> the data is simply not available leading the river to not  
> restart itself on another node?
> 
> On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> The river should be restarted on another node. Maybe you can recreate and  
> explain the states that it happens?
> 
> On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> 
> Hi,
> 
> I am just getting started with elasticsearch and I am running a  
> testcluster with 5 nodes. The indexing is getting done by a river.  
> After one node rebooted the river is not being started automatically  
> on another node. curl -XGET still shows that te river would still be  
> running  
> on the dead node. Am I missing a point?
> 
> Best,  
> Michel.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 12, 2011, 11:00am UTC](https://discuss.elastic.co/t/river-failover/4165/6 "2011-04-12T11:00:20Z")

</div>

When you say delete all storage folders for the database, what database is it? Also, which river are you using, which thread failure are you referring to (which implementation)? Is that a single elasticsearch node scenario now?  
On Tuesday, April 12, 2011 at 1:55 PM, Michel Conrad wrote:

> The issue reappereas on the test cluster, when I do the following steps:
> 
> 1. reset the database by deleting all the storage folders
> 2. starting elasticsearch
> 3. creating the river
> 
> -- now the river runs on one node, in this case 192.168.1.190
> 
> 1. when I kill the this node and immediately restart it, the river is  
> not being restarted, the status of the river stays like this:
> 
> curl [http://127.0.0.1:9200/\_river/dsearch\_all/\_status?pretty=true](http://127.0.0.1:9200/_river/dsearch_all/_status?pretty=true)  
> {  
> "\_index" : "\_river",  
> "\_type" : "dsearch\_all",  
> "\_id" : "\_status",  
> "\_version" : 1, "\_source" :  
> {"ok":true,"node":{"id":"toBTLyVKTIyvcRFHNp-RFg","name":"Marlene  
> Alraune","transport\_address":"inet[/192.168.1.190:9300]"}}  
> }
> 
> additionally if the riverthread on the node is terminated by an  
> exception, the river is not restarted automatically.
> 
> On Tue, Mar 29, 2011 at 1:30 AM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > No, defaults should work and support for river failover... . When you have  
> > more info, ping back.
> > 
> > On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:
> > 
> > I am currently trying to recreate the issue. Could it be that it has  
> > something to do that, as I am using the default  
> > configuration without reconfiguring replication or sharding, a part of  
> > the data is simply not available leading the river to not  
> > restart itself on another node?
> > 
> > On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > 
> > The river should be restarted on another node. Maybe you can recreate and  
> > explain the states that it happens?
> > 
> > On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> > 
> > Hi,
> > 
> > I am just getting started with elasticsearch and I am running a  
> > testcluster with 5 nodes. The indexing is getting done by a river.  
> > After one node rebooted the river is not being started automatically  
> > on another node. curl -XGET still shows that te river would still be  
> > running  
> > on the dead node. Am I missing a point?
> > 
> > Best,  
> > Michel.

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 12, 2011, 11:20am UTC](https://discuss.elastic.co/t/river-failover/4165/7 "2011-04-12T11:20:54Z")

</div>

I have a 5 server scenario to test with, am using a river I wrote to  
import data from a cassandra database and I currently use  
elasticsearch 0.15.2.

When I mean reset the database, I am deleting the folders where  
elasticsearch stores its data (I am using a filebased storage type at  
the moment).

After restarting the cluster, the health goes to green, but the river  
is not being started on any node.

On Tue, Apr 12, 2011 at 1:00 PM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> When you say delete all storage folders for the database, what database is  
> it? Also, which river are you using, which thread failure are you referring  
> to (which implementation)? Is that a single elasticsearch node scenario now?
> 
> On Tuesday, April 12, 2011 at 1:55 PM, Michel Conrad wrote:
> 
> The issue reappereas on the test cluster, when I do the following steps:
> 
> 1. reset the database by deleting all the storage folders
> 2. starting elasticsearch
> 3. creating the river
> 
> -- now the river runs on one node, in this case 192.168.1.190
> 
> 1. when I kill the this node and immediately restart it, the river is  
> not being restarted, the status of the river stays like this:
> 
> curl [http://127.0.0.1:9200/\_river/dsearch\_all/\_status?pretty=true](http://127.0.0.1:9200/_river/dsearch_all/_status?pretty=true)  
> {  
> "\_index" : "\_river",  
> "\_type" : "dsearch\_all",  
> "\_id" : "\_status",  
> "\_version" : 1, "\_source" :  
> {"ok":true,"node":{"id":"toBTLyVKTIyvcRFHNp-RFg","name":"Marlene  
> Alraune","transport\_address":"inet[/192.168.1.190:9300]"}}  
> }
> 
> additionally if the riverthread on the node is terminated by an  
> exception, the river is not restarted automatically.
> 
> On Tue, Mar 29, 2011 at 1:30 AM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> No, defaults should work and support for river failover... . When you have  
> more info, ping back.
> 
> On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:
> 
> I am currently trying to recreate the issue. Could it be that it has  
> something to do that, as I am using the default  
> configuration without reconfiguring replication or sharding, a part of  
> the data is simply not available leading the river to not  
> restart itself on another node?
> 
> On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> The river should be restarted on another node. Maybe you can recreate and  
> explain the states that it happens?
> 
> On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> 
> Hi,
> 
> I am just getting started with elasticsearch and I am running a  
> testcluster with 5 nodes. The indexing is getting done by a river.  
> After one node rebooted the river is not being started automatically  
> on another node. curl -XGET still shows that te river would still be  
> running  
> on the dead node. Am I missing a point?
> 
> Best,  
> Michel.

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 12, 2011, 12:41pm UTC](https://discuss.elastic.co/t/river-failover/4165/8 "2011-04-12T12:41:37Z")

</div>

Here is some additional information from the logfile:

[2011-04-12 14:33:52,321][WARN][river] [Smartship  
Friday] failed to get \_meta from [trendiction]/[dsearch\_all]  
org.elasticsearch.cluster.block.ClusterBlockException: blocked by:  
[1/state not recovered / initialized];  
at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:131)  
at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedRaiseException(ClusterBlocks.java:115)  
at org.elasticsearch.action.get.TransportGetAction.checkBlock(TransportGetAction.java:85)  
at org.elasticsearch.action.get.TransportGetAction.checkBlock(TransportGetAction.java:62)  
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.(TransportShardSingleOperationAction.java:109)  
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.(TransportShardSingleOperationAction.java:88)  
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:69)  
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:45)  
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)  
at org.elasticsearch.client.node.NodeClient.get(NodeClient.java:152)  
at org.elasticsearch.client.action.get.GetRequestBuilder.doExecute(GetRequestBuilder.java:109)  
at org.elasticsearch.client.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:56)  
at org.elasticsearch.river.RiversService$ApplyRivers.riverClusterChanged(RiversService.java:212)  
at org.elasticsearch.river.cluster.RiverClusterService$1.run(RiverClusterService.java:126)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
Connection to 192.168.1.195 closed.

On Tue, Apr 12, 2011 at 1:20 PM, Michel Conrad  
[michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote:

> I have a 5 server scenario to test with, am using a river I wrote to  
> import data from a cassandra database and I currently use  
> elasticsearch 0.15.2.
> 
> When I mean reset the database, I am deleting the folders where  
> elasticsearch stores its data (I am using a filebased storage type at  
> the moment).
> 
> After restarting the cluster, the health goes to green, but the river  
> is not being started on any node.
> 
> On Tue, Apr 12, 2011 at 1:00 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > When you say delete all storage folders for the database, what database is  
> > it? Also, which river are you using, which thread failure are you referring  
> > to (which implementation)? Is that a single elasticsearch node scenario now?
> > 
> > On Tuesday, April 12, 2011 at 1:55 PM, Michel Conrad wrote:
> > 
> > The issue reappereas on the test cluster, when I do the following steps:
> > 
> > 1. reset the database by deleting all the storage folders
> > 2. starting elasticsearch
> > 3. creating the river
> > 
> > -- now the river runs on one node, in this case 192.168.1.190
> > 
> > 1. when I kill the this node and immediately restart it, the river is  
> > not being restarted, the status of the river stays like this:
> > 
> > curl [http://127.0.0.1:9200/\_river/dsearch\_all/\_status?pretty=true](http://127.0.0.1:9200/_river/dsearch_all/_status?pretty=true)  
> > {  
> > "\_index" : "\_river",  
> > "\_type" : "dsearch\_all",  
> > "\_id" : "\_status",  
> > "\_version" : 1, "\_source" :  
> > {"ok":true,"node":{"id":"toBTLyVKTIyvcRFHNp-RFg","name":"Marlene  
> > Alraune","transport\_address":"inet[/192.168.1.190:9300]"}}  
> > }
> > 
> > additionally if the riverthread on the node is terminated by an  
> > exception, the river is not restarted automatically.
> > 
> > On Tue, Mar 29, 2011 at 1:30 AM, Shay Banon  
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > 
> > No, defaults should work and support for river failover... . When you have  
> > more info, ping back.
> > 
> > On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:
> > 
> > I am currently trying to recreate the issue. Could it be that it has  
> > something to do that, as I am using the default  
> > configuration without reconfiguring replication or sharding, a part of  
> > the data is simply not available leading the river to not  
> > restart itself on another node?
> > 
> > On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > 
> > The river should be restarted on another node. Maybe you can recreate and  
> > explain the states that it happens?
> > 
> > On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> > 
> > Hi,
> > 
> > I am just getting started with elasticsearch and I am running a  
> > testcluster with 5 nodes. The indexing is getting done by a river.  
> > After one node rebooted the river is not being started automatically  
> > on another node. curl -XGET still shows that te river would still be  
> > running  
> > on the dead node. Am I missing a point?
> > 
> > Best,  
> > Michel.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 12, 2011, 1:23pm UTC](https://discuss.elastic.co/t/river-failover/4165/9 "2011-04-12T13:23:14Z")

</div>

If you have a custom river, then you need to make sure to handle cases of "river thread" existing and make sure it doesn't.

The log file entry you posted should not cause the river not to start. Its hard for me to help you since I can't run your river (and no, I am not going to run it). Can you recreate it from a clean install of elasticsearch using the "dummy" river?  
On Tuesday, April 12, 2011 at 3:41 PM, Michel Conrad wrote:

> Here is some additional information from the logfile:
> 
> [2011-04-12 14:33:52,321][WARN][river] [Smartship  
> Friday] failed to get \_meta from [trendiction]/[dsearch\_all]  
> org.elasticsearch.cluster.block.ClusterBlockException: blocked by:  
> [1/state not recovered / initialized];  
> at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:131)  
> at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedRaiseException(ClusterBlocks.java:115)  
> at org.elasticsearch.action.get.TransportGetAction.checkBlock(TransportGetAction.java:85)  
> at org.elasticsearch.action.get.TransportGetAction.checkBlock(TransportGetAction.java:62)  
> at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.(TransportShardSingleOperationAction.java:109)  
> at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.(TransportShardSingleOperationAction.java:88)  
> at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:69)  
> at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:45)  
> at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)  
> at org.elasticsearch.client.node.NodeClient.get(NodeClient.java:152)  
> at org.elasticsearch.client.action.get.GetRequestBuilder.doExecute(GetRequestBuilder.java:109)  
> at org.elasticsearch.client.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:56)  
> at org.elasticsearch.river.RiversService$ApplyRivers.riverClusterChanged(RiversService.java:212)  
> at org.elasticsearch.river.cluster.RiverClusterService$1.run(RiverClusterService.java:126)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Connection to 192.168.1.195 closed.
> 
> On Tue, Apr 12, 2011 at 1:20 PM, Michel Conrad  
> [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote:
> 
> > I have a 5 server scenario to test with, am using a river I wrote to  
> > import data from a cassandra database and I currently use  
> > elasticsearch 0.15.2.
> > 
> > When I mean reset the database, I am deleting the folders where  
> > elasticsearch stores its data (I am using a filebased storage type at  
> > the moment).
> > 
> > After restarting the cluster, the health goes to green, but the river  
> > is not being started on any node.
> > 
> > On Tue, Apr 12, 2011 at 1:00 PM, Shay Banon  
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > 
> > > When you say delete all storage folders for the database, what database is  
> > > it? Also, which river are you using, which thread failure are you referring  
> > > to (which implementation)? Is that a single elasticsearch node scenario now?
> > > 
> > > On Tuesday, April 12, 2011 at 1:55 PM, Michel Conrad wrote:
> > > 
> > > The issue reappereas on the test cluster, when I do the following steps:
> > > 
> > > 1. reset the database by deleting all the storage folders
> > > 2. starting elasticsearch
> > > 3. creating the river
> > > 
> > > -- now the river runs on one node, in this case 192.168.1.190
> > > 
> > > 1. when I kill the this node and immediately restart it, the river is  
> > > not being restarted, the status of the river stays like this:
> > > 
> > > curl [http://127.0.0.1:9200/\_river/dsearch\_all/\_status?pretty=true](http://127.0.0.1:9200/_river/dsearch_all/_status?pretty=true)  
> > > {  
> > > "\_index" : "\_river",  
> > > "\_type" : "dsearch\_all",  
> > > "\_id" : "\_status",  
> > > "\_version" : 1, "\_source" :  
> > > {"ok":true,"node":{"id":"toBTLyVKTIyvcRFHNp-RFg","name":"Marlene  
> > > Alraune","transport\_address":"inet[/192.168.1.190:9300]"}}  
> > > }
> > > 
> > > additionally if the riverthread on the node is terminated by an  
> > > exception, the river is not restarted automatically.
> > > 
> > > On Tue, Mar 29, 2011 at 1:30 AM, Shay Banon  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > > 
> > > No, defaults should work and support for river failover... . When you have  
> > > more info, ping back.
> > > 
> > > On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:
> > > 
> > > I am currently trying to recreate the issue. Could it be that it has  
> > > something to do that, as I am using the default  
> > > configuration without reconfiguring replication or sharding, a part of  
> > > the data is simply not available leading the river to not  
> > > restart itself on another node?
> > > 
> > > On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > > 
> > > The river should be restarted on another node. Maybe you can recreate and  
> > > explain the states that it happens?
> > > 
> > > On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> > > 
> > > Hi,
> > > 
> > > I am just getting started with elasticsearch and I am running a  
> > > testcluster with 5 nodes. The indexing is getting done by a river.  
> > > After one node rebooted the river is not being started automatically  
> > > on another node. curl -XGET still shows that te river would still be  
> > > running  
> > > on the dead node. Am I missing a point?
> > > 
> > > Best,  
> > > Michel.

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 12, 2011, 1:26pm UTC](https://discuss.elastic.co/t/river-failover/4165/10 "2011-04-12T13:26:16Z")

</div>

Thanks for the fast reply, I will try it out with the dummy river and  
post the results.

On Tue, Apr 12, 2011 at 3:23 PM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> If you have a custom river, then you need to make sure to handle cases of  
> "river thread" existing and make sure it doesn't.  
> The log file entry you posted should not cause the river not to start. Its  
> hard for me to help you since I can't run your river (and no, I am not going  
> to run it). Can you recreate it from a clean install of elasticsearch using  
> the "dummy" river?
> 
> On Tuesday, April 12, 2011 at 3:41 PM, Michel Conrad wrote:
> 
> Here is some additional information from the logfile:
> 
> [2011-04-12 14:33:52,321][WARN][river]] [Smartship  
> Friday] failed to get \_meta from [trendiction]/[dsearch\_all]  
> org.elasticsearch.cluster.block.ClusterBlockException: blocked by:  
> [1/state not recovered / initialized];  
> at  
> org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:131)  
> at  
> org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedRaiseException(ClusterBlocks.java:115)  
> at  
> org.elasticsearch.action.get.TransportGetAction.checkBlock(TransportGetAction.java:85)  
> at  
> org.elasticsearch.action.get.TransportGetAction.checkBlock(TransportGetAction.java:62)  
> at  
> org.elasticsearch.action.support.single.shard.TransportShardSingleeOperationAction$AsyncSingleAction.(TransportShardSingleOperationAction.java:109)  
> at  
> org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.(TransportShardSingleOperationAction.java:88)  
> at  
> org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:69)  
> at  
> org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:45)  
> at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)  
> at org.elasticsearch.client.node.NodeClient.get(NodeClient.java:152)  
> at  
> org.elasticsearch.client.action.get.GetRequestBuilder.doExecute(GetRequestBuilder.java:109)  
> at  
> org.elasticsearch.client.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:56)  
> at  
> org.elasticsearch.river.RiversService$ApplyRivers.riverClusterChanged(RiversService.java:212)  
> at  
> org.elasticsearch.river.cluster.RiverClusterService$1.run(RiverClusterService.java:126)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Connection to 192.168.1.195 closed.
> 
> On Tue, Apr 12, 2011 at 1:20 PM, Michel Conrad  
> [michel.conrad@trendiction.com](mailto:michel.conrad@trendiction.com) wrote:
> 
> I have a 5 server scenario to test with, am using a river I wrote to  
> import data from a cassandra database and I currently use  
> elasticsearch 0.15.2.
> 
> When I mean reset the database, I am deleting the folders where  
> elasticsearch stores its data (I am using a filebased storage type at  
> the moment).
> 
> After restarting the cluster, the health goes to green, but the river  
> is not being started on any node.
> 
> On Tue, Apr 12, 2011 at 1:00 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> When you say delete all storage folders for the database, what database is  
> it? Also, which river are you using, which thread failure are you referring  
> to (which implementation)? Is that a single elasticsearch node scenario now?
> 
> On Tuesday, April 12, 2011 at 1:55 PM, Michel Conrad wrote:
> 
> The issue reappereas on the test cluster, when I do the following steps:
> 
> 1. reset the database by deleting all the storage folders
> 2. starting elasticsearch
> 3. creating the river
> 
> -- now the river runs on one node, in this case 192.168.1.190
> 
> 1. when I kill the this node and immediately restart it, the river is  
> not being restarted, the status of the river stays like this:
> 
> curl [http://127.0.0.1:9200/\_river/dsearch\_all/\_status?pretty=true](http://127.0.0.1:9200/_river/dsearch_all/_status?pretty=true)  
> {  
> "\_index" : "\_river",  
> "\_type" : "dsearch\_all",  
> "\_id" : "\_status",  
> "\_version" : 1, "\_source" :  
> {"ok":true,"node":{"id":"toBTLyVKTIyvcRFHNp-RFg","name":"Marlene  
> Alraune","transport\_address":"inet[/192.168.1.190:9300]"}}  
> }
> 
> additionally if the riverthread on the node is terminated by an  
> exception, the river is not restarted automatically.
> 
> On Tue, Mar 29, 2011 at 1:30 AM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> No, defaults should work and support for river failover... . When you have  
> more info, ping back.
> 
> On Monday, March 28, 2011 at 1:07 PM, Michel Conrad wrote:
> 
> I am currently trying to recreate the issue. Could it be that it has  
> something to do that, as I am using the default  
> configuration without reconfiguring replication or sharding, a part of  
> the data is simply not available leading the river to not  
> restart itself on another node?
> 
> On Mon, Mar 28, 2011 at 12:20 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> The river should be restarted on another node. Maybe you can recreate and  
> explain the states that it happens?
> 
> On Monday, March 28, 2011 at 11:54 AM, Michel Conrad wrote:
> 
> Hi,
> 
> I am just getting started with elasticsearch and I am running a  
> testcluster with 5 nodes. The indexing is getting done by a river.  
> After one node rebooted the river is not being started automatically  
> on another node. curl -XGET still shows that te river would still be  
> running  
> on the dead node. Am I missing a point?
> 
> Best,  
> Michel.

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 12, 2011, 2:26pm UTC](https://discuss.elastic.co/t/river-failover/4165/11 "2011-04-12T14:26:14Z")

</div>

Even with the dummy river the failover is not working in my case. The  
logfile is from the master node.  
After starting the cluster ths river is run correctly. The logfile  
goes up to this line:

- [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start

When I kill the node running the river, the river is not restarted an  
another node, but instead some errors appear. I am running only the  
dummyriver  
on the cluster.

[2011-04-12 16:13:28,990][INFO][node] [Cody  
Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
initializing ...  
[2011-04-12 16:13:28,995][INFO][plugins] [Cody  
Mushumanski gun Man aka: the hunter] loaded [river-trendiction]  
[2011-04-12 16:13:30,027][WARN][indices] [Cody  
Mushumanski gun Man aka: the hunter] lucene default FieldCache is  
used, not enabling eager reader based cache eviction  
[2011-04-12 16:13:30,116][INFO][node] [Cody  
Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
initialized  
[2011-04-12 16:13:30,116][INFO][node] [Cody  
Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
starting ...  
[2011-04-12 16:13:30,225][INFO][transport] [Cody  
Mushumanski gun Man aka: the hunter] bound\_address  
{inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address  
{inet[/192.168.1.190:9300]}  
[2011-04-12 16:13:33,256][INFO][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] new\_master [Cody Mushumanski gun  
Man aka: the hunter][FG6YrkLqTTqwWxVnA99Xlg][inet[/192.168.1.190:9300]],  
reason: zen-disco-join (elected\_as\_master)  
[2011-04-12 16:13:33,270][INFO][discovery] [Cody  
Mushumanski gun Man aka: the hunter]  
trendictionsearch/FG6YrkLqTTqwWxVnA99Xlg  
[2011-04-12 16:13:33,349][INFO][http] [Cody  
Mushumanski gun Man aka: the hunter] bound\_address  
{inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address  
{inet[/192.168.1.190:9200]}  
[2011-04-12 16:13:33,350][INFO][node] [Cody  
Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
started  
[2011-04-12 16:13:34,986][INFO][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] added {[Connors,  
Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
zen-disco-receive(join from node[[Connors,  
Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]])  
[2011-04-12 16:13:35,964][INFO][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] added  
{[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]],},  
reason: zen-disco-receive(join from  
node[[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]])  
[2011-04-12 16:13:37,146][INFO][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] added  
{[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]],},  
reason: zen-disco-receive(join from  
node[[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]]])  
[2011-04-12 16:13:38,116][INFO][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] added  
{[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]],},  
reason: zen-disco-receive(join from  
node[[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]])  
[2011-04-12 16:13:38,700][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [\_river] creating index, cause  
[gateway], shards [1]/[2], mappings [dsearch\_all, dummy\_river]  
[2011-04-12 16:13:38,870][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] creating  
index, cause [gateway], shards [5]/[2], mappings [content]  
[2011-04-12 16:13:39,041][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] creating  
index, cause [gateway], shards [5]/[2], mappings [content]  
[2011-04-12 16:13:39,178][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] creating  
index, cause [gateway], shards [5]/[2], mappings [content]  
[2011-04-12 16:13:39,294][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all] creating index,  
cause [gateway], shards [5]/[2], mappings [dsearch\_all]  
[2011-04-12 16:13:39,303][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [\_river] created and added to  
cluster\_state  
[2011-04-12 16:13:39,311][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] created and  
added to cluster\_state  
[2011-04-12 16:13:39,324][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] created and  
added to cluster\_state  
[2011-04-12 16:13:39,334][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] created and  
added to cluster\_state  
[2011-04-12 16:13:39,345][INFO][cluster.metadata] [Cody  
Mushumanski gun Man aka: the hunter] [dsearch\_all] created and added  
to cluster\_state  
[2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] create  
[2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start

[2011-04-12 16:14:40,632][WARN][river] [Cody  
Mushumanski gun Man aka: the hunter] failed to create river  
[dummy][dummy\_river]  
org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
shardIt, [1] active : Timeout waiting for [1m], request: index  
{[\_river][dummy\_river][\_status],  
source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
Mushumanski gun Man aka: the  
hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
[2011-04-12 16:15:40,138][WARN][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] failed to reconnect to node  
[Connors, Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]  
org.elasticsearch.transport.ConnectTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] connect\_timeout[30s]  
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:512)  
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:473)  
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)  
at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:301)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
Caused by: java.net.ConnectException: Connection refused  
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)  
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)  
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)  
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)  
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
... 3 more  
[2011-04-12 16:15:40,635][WARN][river] [Cody  
Mushumanski gun Man aka: the hunter] failed to write failed status for  
river creation  
org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
shardIt, [1] active : Timeout waiting for [1m], request: index  
{[\_river][dummy\_river][\_status],  
source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
Mushumanski gun Man aka: the  
hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
[2011-04-12 16:15:40,637][WARN][transport] [Cody  
Mushumanski gun Man aka: the hunter] Received response for a request  
that has timed out, sent [120034ms] ago, timed out [90034ms] ago,  
action [/cluster/nodes/indices/shard/store/node], node  
[[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]], id  
[234]  
[2011-04-12 16:15:40,650][WARN][transport] [Cody  
Mushumanski gun Man aka: the hunter] Received response for a request  
that has timed out, sent [90011ms] ago, timed out [60011ms] ago,  
action [/cluster/nodes/indices/shard/store/node], node  
[[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]], id  
[398]  
[2011-04-12 16:15:40,659][DEBUG][action.search.type] [Cody  
Mushumanski gun Man aka: the hunter] [2] Failed to execute fetch phase  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][search/phase/fetch/id]  
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:318)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:187)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:164)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$600(TransportSearchDfsQueryThenFetchAction.java:63)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:145)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:140)  
at org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:175)  
at org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:168)  
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)  
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:102)  
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)  
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)  
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)  
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)  
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)  
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)  
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)  
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)  
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)  
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
[Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
... 32 more  
[2011-04-12 16:15:40,670][WARN][search.action] [Cody  
Mushumanski gun Man aka: the hunter] Failed to send release search  
context  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][search/freeContext]  
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
at org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:95)  
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:319)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.finishHim(TransportSearchDfsQueryThenFetchAction.java:254)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$1300(TransportSearchDfsQueryThenFetchAction.java:63)  
at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$6.onFailure(TransportSearchDfsQueryThenFetchAction.java:242)  
at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleException(SearchServiceTransportAction.java:329)  
at org.elasticsearch.transport.TransportService$2.run(TransportService.java:197)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
[Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
... 11 more  
[2011-04-12 16:15:40,831][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,957][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,968][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,978][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,982][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,985][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,987][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,990][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,993][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:40,995][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:41,001][WARN][gateway.local] [Cody  
Mushumanski gun Man aka: the hunter] failures when trying to list  
started shards on nodes:  
-\> org.elasticsearch.action.FailedNodeException: Failed node  
[T5cWIcpXTIehneRXTALiLQ];  
org.elasticsearch.transport.SendRequestTransportException: [Connors,  
Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
Curtis][inet[/192.168.1.192:9300]] Node not connected  
[2011-04-12 16:15:41,004][INFO][cluster.service] [Cody  
Mushumanski gun Man aka: the hunter] removed {[Connors,  
Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
zen-disco-node\_left([Connors,  
Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]])

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 12, 2011, 3:53pm UTC](https://discuss.elastic.co/t/river-failover/4165/12 "2011-04-12T15:53:04Z")

</div>

Can you list the exact steps you use to recreate it, including what to start, how many nodes to start, configuration, and so on.  
On Tuesday, April 12, 2011 at 5:26 PM, Michel Conrad wrote:

> Even with the dummy river the failover is not working in my case. The  
> logfile is from the master node.  
> After starting the cluster ths river is run correctly. The logfile  
> goes up to this line:
> 
> - [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> 
> When I kill the node running the river, the river is not restarted an  
> another node, but instead some errors appear. I am running only the  
> dummyriver  
> on the cluster.
> 
> [2011-04-12 16:13:28,990][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> initializing ...  
> [2011-04-12 16:13:28,995][INFO][plugins] [Cody  
> Mushumanski gun Man aka: the hunter] loaded [river-trendiction]  
> [2011-04-12 16:13:30,027][WARN][indices] [Cody  
> Mushumanski gun Man aka: the hunter] lucene default FieldCache is  
> used, not enabling eager reader based cache eviction  
> [2011-04-12 16:13:30,116][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> initialized  
> [2011-04-12 16:13:30,116][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> starting ...  
> [2011-04-12 16:13:30,225][INFO][transport] [Cody  
> Mushumanski gun Man aka: the hunter] bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address  
> {inet[/192.168.1.190:9300]}  
> [2011-04-12 16:13:33,256][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] new\_master [Cody Mushumanski gun  
> Man aka: the hunter][FG6YrkLqTTqwWxVnA99Xlg][inet[/192.168.1.190:9300]],  
> reason: zen-disco-join (elected\_as\_master)  
> [2011-04-12 16:13:33,270][INFO][discovery] [Cody  
> Mushumanski gun Man aka: the hunter]  
> trendictionsearch/FG6YrkLqTTqwWxVnA99Xlg  
> [2011-04-12 16:13:33,349][INFO][http] [Cody  
> Mushumanski gun Man aka: the hunter] bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address  
> {inet[/192.168.1.190:9200]}  
> [2011-04-12 16:13:33,350][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> started  
> [2011-04-12 16:13:34,986][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added {[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> zen-disco-receive(join from node[[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]])  
> [2011-04-12 16:13:35,964][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]])  
> [2011-04-12 16:13:37,146][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]]])  
> [2011-04-12 16:13:38,116][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]])  
> [2011-04-12 16:13:38,700][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [\_river] creating index, cause  
> [gateway], shards [1]/[2], mappings [dsearch\_all, dummy\_river]  
> [2011-04-12 16:13:38,870][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,041][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,178][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,294][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all] creating index,  
> cause [gateway], shards [5]/[2], mappings [dsearch\_all]  
> [2011-04-12 16:13:39,303][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [\_river] created and added to  
> cluster\_state  
> [2011-04-12 16:13:39,311][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,324][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,334][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,345][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all] created and added  
> to cluster\_state  
> [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] create  
> [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> 
> [2011-04-12 16:14:40,632][WARN][river] [Cody  
> Mushumanski gun Man aka: the hunter] failed to create river  
> [dummy][dummy\_river]  
> org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> shardIt, [1] active : Timeout waiting for [1m], request: index  
> {[\_river][dummy\_river][\_status],  
> source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> Mushumanski gun Man aka: the  
> hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-04-12 16:15:40,138][WARN][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] failed to reconnect to node  
> [Connors, Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]  
> org.elasticsearch.transport.ConnectTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] connect\_timeout[30s]  
> at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:512)  
> at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:473)  
> at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)  
> at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:301)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: java.net.ConnectException: Connection refused  
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> ... 3 more  
> [2011-04-12 16:15:40,635][WARN][river] [Cody  
> Mushumanski gun Man aka: the hunter] failed to write failed status for  
> river creation  
> org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> shardIt, [1] active : Timeout waiting for [1m], request: index  
> {[\_river][dummy\_river][\_status],  
> source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> Mushumanski gun Man aka: the  
> hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-04-12 16:15:40,637][WARN][transport] [Cody  
> Mushumanski gun Man aka: the hunter] Received response for a request  
> that has timed out, sent [120034ms] ago, timed out [90034ms] ago,  
> action [/cluster/nodes/indices/shard/store/node], node  
> [[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]], id  
> [234]  
> [2011-04-12 16:15:40,650][WARN][transport] [Cody  
> Mushumanski gun Man aka: the hunter] Received response for a request  
> that has timed out, sent [90011ms] ago, timed out [60011ms] ago,  
> action [/cluster/nodes/indices/shard/store/node], node  
> [[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]], id  
> [398]  
> [2011-04-12 16:15:40,659][DEBUG][action.search.type] [Cody  
> Mushumanski gun Man aka: the hunter] [2] Failed to execute fetch phase  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][search/phase/fetch/id]  
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
> at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:318)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:187)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:164)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$600(TransportSearchDfsQueryThenFetchAction.java:63)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:145)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:140)  
> at org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:175)  
> at org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:168)  
> at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)  
> at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:102)  
> at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)  
> at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)  
> at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)  
> at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)  
> at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)  
> at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
> at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)  
> at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)  
> at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)  
> at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
> at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> ... 32 more  
> [2011-04-12 16:15:40,670][WARN][search.action] [Cody  
> Mushumanski gun Man aka: the hunter] Failed to send release search  
> context  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][search/freeContext]  
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
> at org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:95)  
> at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:319)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.finishHim(TransportSearchDfsQueryThenFetchAction.java:254)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$1300(TransportSearchDfsQueryThenFetchAction.java:63)  
> at org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$6.onFailure(TransportSearchDfsQueryThenFetchAction.java:242)  
> at org.elasticsearch.search.action.SearchServiceTransportAction$8.handleException(SearchServiceTransportAction.java:329)  
> at org.elasticsearch.transport.TransportService$2.run(TransportService.java:197)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
> at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> ... 11 more  
> [2011-04-12 16:15:40,831][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,957][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,968][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,978][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,982][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,985][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,987][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,990][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,993][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,995][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:41,001][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:41,004][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] removed {[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> zen-disco-node\_left([Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]])

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 12, 2011, 4:46pm UTC](https://discuss.elastic.co/t/river-failover/4165/13 "2011-04-12T16:46:11Z")

</div>

I start 5 nodes using the following configuration.  
cluster:  
name: search  
path:  
logs: /var/log/elasticsearch/  
work: /hd1/elasticsearch\_work/  
data: /hd1/elasticsearch\_data/  
index:  
number\_of\_replicas: 2  
gateway:  
recover\_after\_nodes: 3  
recover\_after\_time: 5m  
expected\_nodes: 5

after the health is green I create the river:  
curl -XPUT '[http://127.0.0.1:9200/\_river/dummy\_river/\_meta](http://127.0.0.1:9200/_river/dummy_river/_meta)' -d  
'{"type":"dummy"}'

the river starts on a node. I get the status by: curl -XGET  
[http://127.0.0.1:9200/\_river/dummy\_river/\_status?pretty=true](http://127.0.0.1:9200/_river/dummy_river/_status?pretty=true)

I kill the node where the river runs by:  
pgrep -f elasticsearch | xargs kill

Then the errors from the previous email appear and the river is not  
automatically restarted.  
The status of the river stays the way is was.

On Tue, Apr 12, 2011 at 5:53 PM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> Can you list the exact steps you use to recreate it, including what to  
> start, how many nodes to start, configuration, and so on.
> 
> On Tuesday, April 12, 2011 at 5:26 PM, Michel Conrad wrote:
> 
> Even with the dummy river the failover is not working in my case. The  
> logfile is from the master node.  
> After starting the cluster ths river is run correctly. The logfile  
> goes up to this line:
> 
> - [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> 
> When I kill the node running the river, the river is not restarted an  
> another node, but instead some errors appear. I am running only the  
> dummyriver  
> on the cluster.
> 
> [2011-04-12 16:13:28,990][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> initializing ...  
> [2011-04-12 16:13:28,995][INFO][plugins] [Cody  
> Mushumanski gun Man aka: the hunter] loaded [river-trendiction]  
> [2011-04-12 16:13:30,027][WARN][indices] [Cody  
> Mushumanski gun Man aka: the hunter] lucene default FieldCache is  
> used, not enabling eager reader based cache eviction  
> [2011-04-12 16:13:30,116][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> initialized  
> [2011-04-12 16:13:30,116][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> starting ...  
> [2011-04-12 16:13:30,225][INFO][transport] [Cody  
> Mushumanski gun Man aka: the hunter] bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address  
> {inet[/192.168.1.190:9300]}  
> [2011-04-12 16:13:33,256][INFO][cluster.service]] [Cody  
> Mushumanski gun Man aka: the hunter] new\_master [Cody Mushumanski gun  
> Man aka: the hunter][FG6YrkLqTTqwWxVnA99Xlg][inet[/192.168.1.190:9300]],  
> reason: zen-disco-join (elected\_as\_master)  
> [2011-04-12 16:13:33,270][INFO][discovery] [Cody  
> Mushumanski gun Man aka: the hunter]  
> trendictionsearch/FG6YrkLqTTqwWxVnA99Xlg  
> [2011-04-12 16:13:33,349][INFO][http] [Cody  
> Mushumanski gun Man aka: the hunter] bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address  
> {inet[/192.168.1.190:9200]}  
> [2011-04-12 16:13:33,350][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> started  
> [2011-04-12 16:13:34,986][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added {[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> zen-disco-receive(join from node[[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]])  
> [2011-04-12 16:13:35,964][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]])  
> [2011-04-12 16:13:37,146][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]]])  
> [2011-04-12 16:13:38,116][INFO][cluster.service]] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]])  
> [2011-04-12 16:13:38,700][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [\_river] creating index, cause  
> [gateway], shards [1]/[2], mappings [dsearch\_all, dummy\_river]  
> [2011-04-12 16:13:38,870][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,041][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,178][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,294][INFO][cluster.metadata]] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all] creating index,  
> cause [gateway], shards [5]/[2], mappings [dsearch\_all]  
> [2011-04-12 16:13:39,303][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [\_river] created and added to  
> cluster\_state  
> [2011-04-12 16:13:39,311][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,324][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,334][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,345][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all] created and added  
> to cluster\_state  
> [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] create  
> [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> 
> [2011-04-12 16:14:40,632][WARN][river] [Cody  
> Mushumanski gun Man aka: the hunter] failed to create river  
> [dummy][dummy\_river]  
> org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> shardIt, [1] active : Timeout waiting for [1m], request: index  
> {[\_river][dummy\_river][\_status],  
> source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> Mushumanski gun Man aka: the  
> hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> at  
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-04-12 16:15:40,138][WARN][cluster.service]] [Cody  
> Mushumanski gun Man aka: the hunter] failed to reconnect to node  
> [Connors, Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]  
> org.elasticsearch.transport.ConnectTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] connect\_timeout[30s]  
> at  
> org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:512)  
> at  
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:473)  
> at  
> org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:301)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: java.net.ConnectException: Connection refused  
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> ... 3 more  
> [[2011-04-12 16:15:40,635][WARN][river]] [Cody  
> Mushumanski gun Man aka: the hunter] failed to write failed status for  
> river creation  
> org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> shardIt, [1] active : Timeout waiting for [1m], request: index  
> {[\_river][dummy\_river][\_status],  
> source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> Mushumanski gun Man aka: the  
> hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> at  
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-04-12 16:15:40,637][WARN][transport]] [Cody  
> Mushumanski gun Man aka: the hunter] Received response for a request  
> that has timed out, sent [120034ms] ago, timed out [90034ms] ago,  
> action [/cluster/nodes/indices/shard/store/node], node  
> [[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]], id  
> [234]  
> [2011-04-12 16:15:40,650][WARN][transport] [Cody  
> Mushumanski gun Man aka: the hunter] Received response for a request  
> that has timed out, sent [90011ms] ago, timed out [60011ms] ago,  
> action [/cluster/nodes/indices/shard/store/node], node  
> [[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]], id  
> [398]  
> [2011-04-12 16:15:40,659][DEBUG][action.search.type] [Cody  
> Mushumanski gun Man aka: the hunter] [2] Failed to execute fetch phase  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][search/phase/fetch/id]  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:318)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:187)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:164)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$600(TransportSearchDfsQueryThenFetchAction.java:63)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:145)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:140)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:175)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:168)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:102)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUppstream(DefaultChannelPipeline.java:545)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)  
> at  
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)  
> at  
> org.elasticsearch.common.netty.handler.codec.frame.FFrameDecoder.callDecode(FrameDecoder.java:299)  
> at  
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$$1.run(DeadLockProofWorker.java:44)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> at  
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTrransport.java:566)  
> at  
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> ... 32 more  
> [2011-04-12 16:15:40,670][WARN][search.action] [Cody  
> Mushumanski gun Man aka: the hunter] Failed to send release search  
> context  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][search/freeContext]  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransporrtService.java:166)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:95)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:319)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.finishHim(TransportSearchDfsQueryThenFetchAction.java:254)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$1300(TransportSearchDfsQueryThenFetchAction.java:63)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFFetchAction$AsyncAction$6.onFailure(TransportSearchDfsQueryThenFetchAction.java:242)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$8.handleException(SearchServiceTransportAction.java:329)  
> at  
> org.elasticsearch.transport.TransportService$2.run(TransportService.java:197)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> at  
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
> at  
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> ... 11 more  
> [[2011-04-12 16:15:40,831][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,957][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,968][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,978][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,982][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FFailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,985][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,987][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,990][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,993][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,995][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:41,001][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:41,004][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] removed {[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> zen-disco-node\_left([Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]])

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 12, 2011, 6:53pm UTC](https://discuss.elastic.co/t/river-failover/4165/14 "2011-04-12T18:53:26Z")

</div>

Great, found the problem, which is more fundamental basically revolving around node failure and reallocating the river to another node. Pushed a fix for it, so you can take master for a spin: [River: Failure to reallocate river to another node on rivers node failure · Issue #850 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/850).

Thanks for sticking around and helping flush out this important bug.  
On Tuesday, April 12, 2011 at 7:46 PM, Michel Conrad wrote:

> I start 5 nodes using the following configuration.  
> cluster:  
> name: search  
> path:  
> logs: /var/log/elasticsearch/  
> work: /hd1/elasticsearch\_work/  
> data: /hd1/elasticsearch\_data/  
> index:  
> number\_of\_replicas: 2  
> gateway:  
> recover\_after\_nodes: 3  
> recover\_after\_time: 5m  
> expected\_nodes: 5
> 
> after the health is green I create the river:  
> curl -XPUT '[http://127.0.0.1:9200/\_river/dummy\_river/\_meta](http://127.0.0.1:9200/_river/dummy_river/_meta)' -d  
> '{"type":"dummy"}'
> 
> the river starts on a node. I get the status by: curl -XGET  
> [http://127.0.0.1:9200/\_river/dummy\_river/\_status?pretty=true](http://127.0.0.1:9200/_river/dummy_river/_status?pretty=true)
> 
> I kill the node where the river runs by:  
> pgrep -f elasticsearch | xargs kill
> 
> Then the errors from the previous email appear and the river is not  
> automatically restarted.  
> The status of the river stays the way is was.
> 
> On Tue, Apr 12, 2011 at 5:53 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> > Can you list the exact steps you use to recreate it, including what to  
> > start, how many nodes to start, configuration, and so on.
> > 
> > On Tuesday, April 12, 2011 at 5:26 PM, Michel Conrad wrote:
> > 
> > Even with the dummy river the failover is not working in my case. The  
> > logfile is from the master node.  
> > After starting the cluster ths river is run correctly. The logfile  
> > goes up to this line:
> > 
> > - [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> > Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> > 
> > When I kill the node running the river, the river is not restarted an  
> > another node, but instead some errors appear. I am running only the  
> > dummyriver  
> > on the cluster.
> > 
> > [2011-04-12 16:13:28,990][INFO][node] [Cody  
> > Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> > initializing ...  
> > [2011-04-12 16:13:28,995][INFO][plugins] [Cody  
> > Mushumanski gun Man aka: the hunter] loaded [river-trendiction]  
> > [2011-04-12 16:13:30,027][WARN][indices] [Cody  
> > Mushumanski gun Man aka: the hunter] lucene default FieldCache is  
> > used, not enabling eager reader based cache eviction  
> > [2011-04-12 16:13:30,116][INFO][node] [Cody  
> > Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> > initialized  
> > [2011-04-12 16:13:30,116][INFO][node] [Cody  
> > Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> > starting ...  
> > [2011-04-12 16:13:30,225][INFO][transport] [Cody  
> > Mushumanski gun Man aka: the hunter] bound\_address  
> > {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address  
> > {inet[/192.168.1.190:9300]}  
> > [2011-04-12 16:13:33,256][INFO][cluster.service]] [Cody  
> > Mushumanski gun Man aka: the hunter] new\_master [Cody Mushumanski gun  
> > Man aka: the hunter][FG6YrkLqTTqwWxVnA99Xlg][inet[/192.168.1.190:9300]],  
> > reason: zen-disco-join (elected\_as\_master)  
> > [2011-04-12 16:13:33,270][INFO][discovery] [Cody  
> > Mushumanski gun Man aka: the hunter]  
> > trendictionsearch/FG6YrkLqTTqwWxVnA99Xlg  
> > [2011-04-12 16:13:33,349][INFO][http] [Cody  
> > Mushumanski gun Man aka: the hunter] bound\_address  
> > {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address  
> > {inet[/192.168.1.190:9200]}  
> > [2011-04-12 16:13:33,350][INFO][node] [Cody  
> > Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> > started  
> > [2011-04-12 16:13:34,986][INFO][cluster.service] [Cody  
> > Mushumanski gun Man aka: the hunter] added {[Connors,  
> > Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> > zen-disco-receive(join from node[[Connors,  
> > Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]])  
> > [2011-04-12 16:13:35,964][INFO][cluster.service] [Cody  
> > Mushumanski gun Man aka: the hunter] added  
> > {[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]],},  
> > reason: zen-disco-receive(join from  
> > node[[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]])  
> > [2011-04-12 16:13:37,146][INFO][cluster.service] [Cody  
> > Mushumanski gun Man aka: the hunter] added  
> > {[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]],},  
> > reason: zen-disco-receive(join from  
> > node[[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]]])  
> > [2011-04-12 16:13:38,116][INFO][cluster.service]] [Cody  
> > Mushumanski gun Man aka: the hunter] added  
> > {[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]],},  
> > reason: zen-disco-receive(join from  
> > node[[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]])  
> > [2011-04-12 16:13:38,700][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [\_river] creating index, cause  
> > [gateway], shards [1]/[2], mappings [dsearch\_all, dummy\_river]  
> > [2011-04-12 16:13:38,870][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] creating  
> > index, cause [gateway], shards [5]/[2], mappings [content]  
> > [2011-04-12 16:13:39,041][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] creating  
> > index, cause [gateway], shards [5]/[2], mappings [content]  
> > [2011-04-12 16:13:39,178][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] creating  
> > index, cause [gateway], shards [5]/[2], mappings [content]  
> > [2011-04-12 16:13:39,294][INFO][cluster.metadata]] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all] creating index,  
> > cause [gateway], shards [5]/[2], mappings [dsearch\_all]  
> > [2011-04-12 16:13:39,303][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [\_river] created and added to  
> > cluster\_state  
> > [2011-04-12 16:13:39,311][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] created and  
> > added to cluster\_state  
> > [2011-04-12 16:13:39,324][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] created and  
> > added to cluster\_state  
> > [2011-04-12 16:13:39,334][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] created and  
> > added to cluster\_state  
> > [2011-04-12 16:13:39,345][INFO][cluster.metadata] [Cody  
> > Mushumanski gun Man aka: the hunter] [dsearch\_all] created and added  
> > to cluster\_state  
> > [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> > Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] create  
> > [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> > Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> > 
> > [2011-04-12 16:14:40,632][WARN][river] [Cody  
> > Mushumanski gun Man aka: the hunter] failed to create river  
> > [dummy][dummy\_river]  
> > org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> > shardIt, [1] active : Timeout waiting for [1m], request: index  
> > {[\_river][dummy\_river][\_status],  
> > source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> > Mushumanski gun Man aka: the  
> > hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> > at  
> > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> > at  
> > org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > [2011-04-12 16:15:40,138][WARN][cluster.service]] [Cody  
> > Mushumanski gun Man aka: the hunter] failed to reconnect to node  
> > [Connors, Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]  
> > org.elasticsearch.transport.ConnectTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] connect\_timeout[30s]  
> > at  
> > org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:512)  
> > at  
> > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:473)  
> > at  
> > org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)  
> > at  
> > org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:301)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > Caused by: java.net.ConnectException: Connection refused  
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
> > at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)  
> > at  
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > at  
> > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> > ... 3 more  
> > [[2011-04-12 16:15:40,635][WARN][river]] [Cody  
> > Mushumanski gun Man aka: the hunter] failed to write failed status for  
> > river creation  
> > org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> > shardIt, [1] active : Timeout waiting for [1m], request: index  
> > {[\_river][dummy\_river][\_status],  
> > source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> > Mushumanski gun Man aka: the  
> > hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> > at  
> > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> > at  
> > org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > [2011-04-12 16:15:40,637][WARN][transport]] [Cody  
> > Mushumanski gun Man aka: the hunter] Received response for a request  
> > that has timed out, sent [120034ms] ago, timed out [90034ms] ago,  
> > action [/cluster/nodes/indices/shard/store/node], node  
> > [[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]], id  
> > [234]  
> > [2011-04-12 16:15:40,650][WARN][transport] [Cody  
> > Mushumanski gun Man aka: the hunter] Received response for a request  
> > that has timed out, sent [90011ms] ago, timed out [60011ms] ago,  
> > action [/cluster/nodes/indices/shard/store/node], node  
> > [[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]], id  
> > [398]  
> > [2011-04-12 16:15:40,659][DEBUG][action.search.type] [Cody  
> > Mushumanski gun Man aka: the hunter] [2] Failed to execute fetch phase  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][search/phase/fetch/id]  
> > at  
> > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> > at  
> > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:318)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:187)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:164)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$600(TransportSearchDfsQueryThenFetchAction.java:63)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:145)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:140)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:175)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:168)  
> > at  
> > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)  
> > at  
> > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:102)  
> > at  
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUppstream(DefaultChannelPipeline.java:545)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)  
> > at  
> > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)  
> > at  
> > org.elasticsearch.common.netty.handler.codec.frame.FFrameDecoder.callDecode(FrameDecoder.java:299)  
> > at  
> > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)  
> > at  
> > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
> > at  
> > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> > at  
> > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)  
> > at  
> > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)  
> > at  
> > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> > at  
> > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$$1.run(DeadLockProofWorker.java:44)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> > [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > at  
> > org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTrransport.java:566)  
> > at  
> > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> > at  
> > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> > ... 32 more  
> > [2011-04-12 16:15:40,670][WARN][search.action] [Cody  
> > Mushumanski gun Man aka: the hunter] Failed to send release search  
> > context  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][search/freeContext]  
> > at  
> > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> > at  
> > org.elasticsearch.transport.TransportService.sendRequest(TransporrtService.java:166)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:95)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:319)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.finishHim(TransportSearchDfsQueryThenFetchAction.java:254)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$1300(TransportSearchDfsQueryThenFetchAction.java:63)  
> > at  
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFFetchAction$AsyncAction$6.onFailure(TransportSearchDfsQueryThenFetchAction.java:242)  
> > at  
> > org.elasticsearch.search.action.SearchServiceTransportAction$8.handleException(SearchServiceTransportAction.java:329)  
> > at  
> > org.elasticsearch.transport.TransportService$2.run(TransportService.java:197)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> > [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > at  
> > org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
> > at  
> > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> > at  
> > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> > ... 11 more  
> > [[2011-04-12 16:15:40,831][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,957][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,968][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,978][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,982][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FFailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,985][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,987][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,990][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,993][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:40,995][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:41,001][WARN][gateway.local] [Cody  
> > Mushumanski gun Man aka: the hunter] failures when trying to list  
> > started shards on nodes:  
> > -\> org.elasticsearch.action.FailedNodeException: Failed node  
> > [T5cWIcpXTIehneRXTALiLQ];  
> > org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> > org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> > Curtis][inet[/192.168.1.192:9300]] Node not connected  
> > [2011-04-12 16:15:41,004][INFO][cluster.service] [Cody  
> > Mushumanski gun Man aka: the hunter] removed {[Connors,  
> > Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> > zen-disco-node\_left([Connors,  
> > Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]])

---

<div class="post-metadata">

**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\
**Post date:** [April 13, 2011, 12:50pm UTC](https://discuss.elastic.co/t/river-failover/4165/15 "2011-04-13T12:50:08Z")

</div>

I switched to master and gave it a try. Failover now works like a charm.

Thanks for the quick reaction and the fix.

On Tue, Apr 12, 2011 at 8:53 PM, Shay Banon  
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:

> Great, found the problem, which is more fundamental basically revolving  
> around node failure and reallocating the river to another node. Pushed a fix  
> for it, so you can take master for a  
> spin: [River: Failure to reallocate river to another node on rivers node failure · Issue #850 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/850).  
> Thanks for sticking around and helping flush out this important bug.
> 
> On Tuesday, April 12, 2011 at 7:46 PM, Michel Conrad wrote:
> 
> I start 5 nodes using the following configuration.  
> cluster:  
> name: search  
> path:  
> logs: /var/log/elasticsearch/  
> work: /hd1/elasticsearch\_work/  
> data: /hd1/elasticsearch\_data/  
> index:  
> number\_of\_replicas: 2  
> gateway:  
> recover\_after\_nodes: 3  
> recover\_after\_time: 5m  
> expected\_nodes: 5
> 
> after the health is green I create the river:  
> curl -XPUT '[http://127.0.0.1:9200/\_river/dummy\_river/\_meta](http://127.0.0.1:9200/_river/dummy_river/_meta)' -d  
> '{"type":"dummy"}'
> 
> the river starts on a node. I get the status by: curl -XGET  
> [http://127.0.0.1:9200/\_river/dummy\_river/\_status?pretty=true](http://127.0.0.1:9200/_river/dummy_river/_status?pretty=true)
> 
> I kill the node where the river runs by:  
> pgrep -f elasticsearch | xargs kill
> 
> Then the errors from the previous email appear and the river is not  
> automatically restarted.  
> The status of the river stays the way is was.
> 
> On Tue, Apr 12, 2011 at 5:53 PM, Shay Banon  
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> 
> Can you list the exact steps you use to recreate it, including what to  
> start, how many nodes to start, configuration, and so on.
> 
> On Tuesday, April 12, 2011 at 5:26 PM, Michel Conrad wrote:
> 
> Even with the dummy river the failover is not working in my case. The  
> logfile is from the master node.  
> After starting the cluster ths river is run correctly. The logfile  
> goes up to this line:
> 
> - [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> 
> When I kill the node running the river, the river is not restarted an  
> another node, but instead some errors appear. I am running only the  
> dummyriver  
> on the cluster.
> 
> [2011-04-12 16:13:28,990][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> initializing ...  
> [2011-04-12 16:13:28,995][INFO][plugins] [Cody  
> Mushumanski gun Man aka: the hunter] loaded [river-trendiction]  
> [2011-04-12 16:13:30,027][WARN][indices] [Cody  
> Mushumanski gun Man aka: the hunter] lucene default FieldCache is  
> used, not enabling eager reader based cache eviction  
> [2011-04-12 16:13:30,116][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> initialized  
> [2011-04-12 16:13:30,116][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> starting ...  
> [2011-04-12 16:13:30,225][INFO][transport] [Cody  
> Mushumanski gun Man aka: the hunter] bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address  
> {inet[/192.168.1.190:9300]}  
> [2011-04-12 16:13:33,256][INFO][cluster.service]] [Cody  
> Mushumanski gun Man aka: the hunter] new\_master [Cody Mushumanski gun  
> Man aka: the hunter][FG6YrkLqTTqwWxVnA99Xlg][inet[/192.168.1.190:9300]],  
> reason: zen-disco-join (elected\_as\_master)  
> [2011-04-12 16:13:33,270][INFO][discovery] [Cody  
> Mushumanski gun Man aka: the hunter]  
> trendictionsearch/FG6YrkLqTTqwWxVnA99Xlg  
> [2011-04-12 16:13:33,349][INFO][http] [Cody  
> Mushumanski gun Man aka: the hunter] bound\_address  
> {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address  
> {inet[/192.168.1.190:9200]}  
> [2011-04-12 16:13:33,350][INFO][node] [Cody  
> Mushumanski gun Man aka: the hunter] {elasticsearch/0.15.2}[8729]:  
> started  
> [2011-04-12 16:13:34,986][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added {[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> zen-disco-receive(join from node[[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]])  
> [2011-04-12 16:13:35,964][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]])  
> [2011-04-12 16:13:37,146][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Poison][OWoKy1PKRqKB9yTgfz\_7Eg][inet[/192.168.1.191:9300]]])  
> [2011-04-12 16:13:38,116][INFO][cluster.service]] [Cody  
> Mushumanski gun Man aka: the hunter] added  
> {[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]],},  
> reason: zen-disco-receive(join from  
> node[[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]])  
> [2011-04-12 16:13:38,700][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [\_river] creating index, cause  
> [gateway], shards [1]/[2], mappings [dsearch\_all, dummy\_river]  
> [2011-04-12 16:13:38,870][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,041][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,178][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] creating  
> index, cause [gateway], shards [5]/[2], mappings [content]  
> [2011-04-12 16:13:39,294][INFO][cluster.metadata]] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all] creating index,  
> cause [gateway], shards [5]/[2], mappings [dsearch\_all]  
> [2011-04-12 16:13:39,303][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [\_river] created and added to  
> cluster\_state  
> [2011-04-12 16:13:39,311][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adb] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,324][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3adc] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,334][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all\_3add] created and  
> added to cluster\_state  
> [2011-04-12 16:13:39,345][INFO][cluster.metadata] [Cody  
> Mushumanski gun Man aka: the hunter] [dsearch\_all] created and added  
> to cluster\_state  
> [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] create  
> [2011-04-12 16:13:40,600][INFO][river.dummy] [Cody  
> Mushumanski gun Man aka: the hunter] [dummy][dummy\_river] start
> 
> [2011-04-12 16:14:40,632][WARN][river] [Cody  
> Mushumanski gun Man aka: the hunter] failed to create river  
> [dummy][dummy\_river]  
> org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> shardIt, [1] active : Timeout waiting for [1m], request: index  
> {[\_river][dummy\_river][\_status],  
> source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> Mushumanski gun Man aka: the  
> hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> at  
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-04-12 16:15:40,138][WARN][cluster.service]] [Cody  
> Mushumanski gun Man aka: the hunter] failed to reconnect to node  
> [Connors, Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]]  
> org.elasticsearch.transport.ConnectTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] connect\_timeout[30s]  
> at  
> org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:512)  
> at  
> org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:473)  
> at  
> org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:126)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:301)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: java.net.ConnectException: Connection refused  
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)  
> ... 3 more  
> [[2011-04-12 16:15:40,635][WARN][river]] [Cody  
> Mushumanski gun Man aka: the hunter] failed to write failed status for  
> river creation  
> org.elasticsearch.action.UnavailableShardsException: [\_river][0] [3]  
> shardIt, [1] active : Timeout waiting for [1m], request: index  
> {[\_river][dummy\_river][\_status],  
> source[{"ok":true,"node":{"id":"FG6YrkLqTTqwWxVnA99Xlg","name":"Cody  
> Mushumanski gun Man aka: the  
> hunter","transport\_address":"inet[/192.168.1.190:9300]"}}]}  
> at  
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$3.onTimeout(TransportShardReplicationOperationAction.java:409)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:281)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-04-12 16:15:40,637][WARN][transport]] [Cody  
> Mushumanski gun Man aka: the hunter] Received response for a request  
> that has timed out, sent [120034ms] ago, timed out [90034ms] ago,  
> action [/cluster/nodes/indices/shard/store/node], node  
> [[Zaladane][6pbf3UETTdiuiyHUIk6bwQ][inet[/192.168.1.194:9300]]], id  
> [234]  
> [2011-04-12 16:15:40,650][WARN][transport] [Cody  
> Mushumanski gun Man aka: the hunter] Received response for a request  
> that has timed out, sent [90011ms] ago, timed out [60011ms] ago,  
> action [/cluster/nodes/indices/shard/store/node], node  
> [[Dagger][DN31iuVQRw2MKAD3U5Q1mg][inet[/192.168.1.195:9300]]], id  
> [398]  
> [2011-04-12 16:15:40,659][DEBUG][action.search.type] [Cody  
> Mushumanski gun Man aka: the hunter] [2] Failed to execute fetch phase  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][search/phase/fetch/id]  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:166)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:318)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFFetchAction$AsyncAction.executeFetch(TransportSearchDfsQueryThenFetchAction.java:226)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.innerExecuteFetchPhase(TransportSearchDfsQueryThenFetchAction.java:187)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeFetchPhase(TransportSearchDfsQueryThenFetchAction.java:164)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$600(TransportSearchDfsQueryThenFetchAction.java:63)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:145)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchDfsQueryThenFetchAction.java:140)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:175)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$3.handleResponse(SearchServiceTransportAction.java:168)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:132)  
> at  
> org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:102)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUppstream(DefaultChannelPipeline.java:545)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:302)  
> at  
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)  
> at  
> org.elasticsearch.common.netty.handler.codec.frame.FFrameDecoder.callDecode(FrameDecoder.java:299)  
> at  
> org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)  
> at  
> org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)  
> at  
> org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:274)  
> at  
> org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:261)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$$1.run(DeadLockProofWorker.java:44)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> at  
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTrransport.java:566)  
> at  
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> ... 32 more  
> [2011-04-12 16:15:40,670][WARN][search.action] [Cody  
> Mushumanski gun Man aka: the hunter] Failed to send release search  
> context  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][search/freeContext]  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:194)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransporrtService.java:166)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction.sendFreeContext(SearchServiceTransportAction.java:95)  
> at  
> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.releaseIrrelevantSearchContexts(TransportSearchTypeAction.java:319)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.finishHim(TransportSearchDfsQueryThenFetchAction.java:254)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.access$1300(TransportSearchDfsQueryThenFetchAction.java:63)  
> at  
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFFetchAction$AsyncAction$6.onFailure(TransportSearchDfsQueryThenFetchAction.java:242)  
> at  
> org.elasticsearch.search.action.SearchServiceTransportAction$8.handleException(SearchServiceTransportAction.java:329)  
> at  
> org.elasticsearch.transport.TransportService$2.run(TransportService.java:197)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> Caused by: org.elasticsearch.transport.NodeNotConnectedException:  
> [Connors, Curtis][inet[/192.168.1.192:9300]] Node not connected  
> at  
> org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:566)  
> at  
> org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:424)  
> at  
> org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:179)  
> ... 11 more  
> [[2011-04-12 16:15:40,831][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,957][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,968][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,978][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,982][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FFailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,985][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,987][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,990][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,993][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:40,995][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:41,001][WARN][gateway.local] [Cody  
> Mushumanski gun Man aka: the hunter] failures when trying to list  
> started shards on nodes:  
> -\> org.elasticsearch.action.FailedNodeException: Failed node  
> [T5cWIcpXTIehneRXTALiLQ];  
> org.elasticsearch.transport.SendRequestTransportException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]][/gateway/local/started-shards/node];  
> org.elasticsearch.transport.NodeNotConnectedException: [Connors,  
> Curtis][inet[/192.168.1.192:9300]] Node not connected  
> [2011-04-12 16:15:41,004][INFO][cluster.service] [Cody  
> Mushumanski gun Man aka: the hunter] removed {[Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]],}, reason:  
> zen-disco-node\_left([Connors,  
> Curtis][T5cWIcpXTIehneRXTALiLQ][inet[/192.168.1.192:9300]])

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:08am UTC](https://discuss.elastic.co/t/river-failover/4165/16 "2017-07-06T04:08:24Z")

</div>