# River not being migrated to new node on service restart (ES 0.15.0) **URL:** \ **Category:** Elasticsearch\ **Created:** [February 22, 2011, 11:46am UTC](https://discuss.elastic.co/t/river-not-being-migrated-to-new-node-on-service-restart-es-0-15-0/3959 "2011-02-22T11:46:02Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![charles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles/32/3254_2.png) [@charles](https://discuss.elastic.co/u/charles)\ **Post date:** [February 22, 2011, 11:46am UTC](https://discuss.elastic.co/t/river-not-being-migrated-to-new-node-on-service-restart-es-0-15-0/3959/1 "2011-02-22T11:46:02Z")
Howdy! I created an AMQP river to feed in log events. It successfully establishes a connection, and all works well... until I restart my node. When that happens, the river stays attached to the old node name, rather than being started up anew. Thus, I end in a situation like the following: $ curl [http://127.0.0.1:9200/\_river/logstash\_events/\_status?pretty=true](http://127.0.0.1:9200/_river/logstash_events/_status?pretty=true) { "\_index" : "\_river", "\_type" : "logstash\_events", "\_id" : "\_status", "\_version" : 5, "\_source" : {"ok":true,"node": {"id":"mN0roKFeTOWNUVpq4mQZqg","name":"Captain Fate","transport\_address":"inet[/10.0.2.15:9300]"}} } $ curl [http://127.0.0.1:9200/\_cluster/state?pretty=true](http://127.0.0.1:9200/_cluster/state?pretty=true) ... "nodes" : { "tklLqDDSRseHvn9Eq2V3Zw" : { "name" : "Potts, Virginia "Pepper"", "transport\_address" : "inet[/10.0.2.15:9300]", "attributes" : { } } ... ...notably, the river is still associated with Captain Fate, despite Captain Fate having passed away in favor of Virginia "Pepper" Pots. I can get things unstuck by deleting and recreating the river whenever the condition occurs, but that's a fair sight less than graceful. Is there a better approach? Thanks! ---
**Author:** ![charles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles/32/3254_2.png) [@charles](https://discuss.elastic.co/u/charles)\ **Post date:** [February 22, 2011, 1:08pm UTC](https://discuss.elastic.co/t/river-not-being-migrated-to-new-node-on-service-restart-es-0-15-0/3959/2 "2011-02-22T13:08:15Z")
Actually -- this looks to be trickier than I previously expected: DELETE'ing a river (even restarting after doing so) is not always sufficient to make it re-add'able under the same name, though shifting to a different name works. # netstat --tcp -ep | grep java | grep amqp tcp 0 0 logs.vguest:41506 mon1.vguest:amqp ESTABLISHED elasticsearch 146425 20600/ java $ curl -XDELETE 'localhost:9200/\_river/logstash\_events\_201102220651/ \_meta' {"ok":true,"found":true,"\_index":"\_river","\_type":"logstash\_events\_201102220651","\_id":"\_meta","\_version": 2} # netstat --tcp -ep | grep java | grep amqp tcp 0 0 logs.vguest:41506 mon1.vguest:amqp ESTABLISHED elasticsearch 146425 20600/ java # sv t /service/elasticsearch ## restart the service # netstat --tcp -ep | grep java | grep amqp ## the amqp connection is gone now $ curl -v -XPUT 'localhost:9200/\_river/logstash\_events\_201102220700/ \_meta' -d ' { "type" : "rabbitmq", "rabbitmq" : { "host" : "192.168.123.8", "port" : 5672, "user" : "logstash", "pass" : "logstash", "vhost" : "logstash", "exchange" : "parsed\_logs", "exchange\_type" : "fanout", "exchange\_durable": "true", "queue" : "elasticsearch", "queue\_durable": "true", "routing\_key" : "elasticsearch" }, "index" : { "bulk\_size" : 100, "bulk\_timeout" : "10ms" } }' # netstat --tcp -ep | grep java | grep amqp ## ...but hey, it's still gone... $ curl -v -XPUT 'localhost:9200/\_river/logstash\_events\_201102220702/ \_meta' -d ' { "type" : "rabbitmq", "rabbitmq" : { "host" : "192.168.123.8", "port" : 5672, "user" : "logstash", "pass" : "logstash", "vhost" : "logstash", "exchange" : "parsed\_logs", "exchange\_type" : "fanout", "exchange\_durable": "true", "queue" : "elasticsearch", "queue\_durable": "true", "routing\_key" : "elasticsearch" }, "index" : { "bulk\_size" : 100, "bulk\_timeout" : "10ms" } }' # netstat --tcp -ep | grep java | grep amqp tcp 0 0 logs.vguest:56203 mon1.vguest:amqp ESTABLISHED elasticsearch 148361 21250/ java ...until we created it under a new name, at which time it works fine. ---
**Author:** ![charles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles/32/3254_2.png) [@charles](https://discuss.elastic.co/u/charles)\ **Post date:** [February 22, 2011, 2:07pm UTC](https://discuss.elastic.co/t/river-not-being-migrated-to-new-node-on-service-restart-es-0-15-0/3959/3 "2011-02-22T14:07:19Z")
Downgrading to 0.14.x, behavior is unchanged -- with the exception that I get an error message during startup which at least provides an appearance of making some sense of the situation: 2011-02-22\_13:59:49.62013 [13:59:49,619][WARN] [river.routing] [Ultra-Marine] failed to get/parse \_meta for [logstash\_events\_201102220700] org.elasticsearch.action.NoShardAvailableActionException: [\_river]3] No shard available for [logstash\_events\_201102220700#\_meta] ...frankly, while I'm still not able to have a river persist through cluster restart, getting at least an error message is good for morale. 🙂 On Feb 22, 7:08 am, Charles Duffy [char...@dyfis.net](mailto:char...@dyfis.net) wrote: > Actually -- this looks to be trickier than I previously expected: > DELETE'ing a river (even restarting after doing so) is not always > sufficient to make it re-add'able under the same name, though shifting > to a different name works. > > # netstat --tcp -ep | grep java | grep amqp > > tcp 0 0 logs.vguest:41506 > mon1.vguest:amqp ESTABLISHED elasticsearch 146425 20600/ > java > $ curl -XDELETE 'localhost:9200/\_river/logstash\_events\_201102220651/ > \_meta' > {"ok":true,"found":true,"\_index":"\_river","\_type":"logstash\_events\_20110222 0651","\_id":"\_meta","\_version": > 2} > > # netstat --tcp -ep | grep java | grep amqp > > tcp 0 0 logs.vguest:41506 > mon1.vguest:amqp ESTABLISHED elasticsearch 146425 20600/ > java > > # sv t /service/elasticsearch ## restart the service > > # netstat --tcp -ep | grep java | grep amqp ## the amqp connection is > > gone now > $ curl -v -XPUT 'localhost:9200/\_river/logstash\_events\_201102220700/ > \_meta' -d ' > { > "type" : "rabbitmq", > "rabbitmq" : { > "host" : "192.168.123.8", > "port" : 5672, > "user" : "logstash", > "pass" : "logstash", > "vhost" : "logstash", > "exchange" : "parsed\_logs", > "exchange\_type" : "fanout", > "exchange\_durable": "true", > "queue" : "elasticsearch", > "queue\_durable": "true", > "routing\_key" : "elasticsearch" > }, > "index" : { > "bulk\_size" : 100, > "bulk\_timeout" : "10ms" > }}' > > # netstat --tcp -ep | grep java | grep amqp ## ...but hey, it's still > > gone... > $ curl -v -XPUT 'localhost:9200/\_river/logstash\_events\_201102220702/ > \_meta' -d ' > { > "type" : "rabbitmq", > "rabbitmq" : { > "host" : "192.168.123.8", > "port" : 5672, > "user" : "logstash", > "pass" : "logstash", > "vhost" : "logstash", > "exchange" : "parsed\_logs", > "exchange\_type" : "fanout", > "exchange\_durable": "true", > "queue" : "elasticsearch", > "queue\_durable": "true", > "routing\_key" : "elasticsearch" > }, > "index" : { > "bulk\_size" : 100, > "bulk\_timeout" : "10ms" > }}' > > # netstat --tcp -ep | grep java | grep amqp > > tcp 0 0 logs.vguest:56203 > mon1.vguest:amqp ESTABLISHED elasticsearch 148361 21250/ > java > > ...until we created it under a new name, at which time it works fine. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 22, 2011, 6:47pm UTC](https://discuss.elastic.co/t/river-not-being-migrated-to-new-node-on-service-restart-es-0-15-0/3959/4 "2011-02-22T18:47:11Z")
Heya, Found the problem, it relates to restoring rivers when there in a single node cluster. Fixed it: [River not recovered when using single node after shutdown · Issue #711 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/711). You can work around it by "forcing" a cluster change by either adding another node or creating a dummy index (and deleting it afterwards). Hackish, I know, but this fix will be part of 0.15.1. -shay.banon On Tuesday, February 22, 2011 at 4:07 PM, Charles Duffy wrote: > Downgrading to 0.14.x, behavior is unchanged -- with the exception > that I get an error message during startup which at least provides an > appearance of making some sense of the situation: > > 2011-02-22\_13:59:49.62013 [13:59:49,619][WARN] > [river.routing] [Ultra-Marine] failed to get/parse \_meta > for [logstash\_events\_201102220700] > org.elasticsearch.action.NoShardAvailableActionException: [\_river]3] > No shard available for [logstash\_events\_201102220700#\_meta] > > ...frankly, while I'm still not able to have a river persist through > cluster restart, getting at least an error message is good for > morale. 🙂 > > On Feb 22, 7:08 am, Charles Duffy [char...@dyfis.net](mailto:char...@dyfis.net) wrote: > > > Actually -- this looks to be trickier than I previously expected: > > DELETE'ing a river (even restarting after doing so) is not always > > sufficient to make it re-add'able under the same name, though shifting > > to a different name works. > > > > # netstat --tcp -ep | grep java | grep amqp > > > > tcp 0 0 logs.vguest:41506 > > mon1.vguest:amqp ESTABLISHED elasticsearch 146425 20600/ > > java > > $ curl -XDELETE 'localhost:9200/\_river/logstash\_events\_201102220651/ > > \_meta' > > {"ok":true,"found":true,"\_index":"\_river","\_type":"logstash\_events\_20110222 0651","\_id":"\_meta","\_version": > > 2} > > > > # netstat --tcp -ep | grep java | grep amqp > > > > tcp 0 0 logs.vguest:41506 > > mon1.vguest:amqp ESTABLISHED elasticsearch 146425 20600/ > > java > > > > # sv t /service/elasticsearch ## restart the service > > > > # netstat --tcp -ep | grep java | grep amqp ## the amqp connection is > > > > gone now > > $ curl -v -XPUT 'localhost:9200/\_river/logstash\_events\_201102220700/ > > \_meta' -d ' > > { > > "type" : "rabbitmq", > > "rabbitmq" : { > > "host" : "192.168.123.8", > > "port" : 5672, > > "user" : "logstash", > > "pass" : "logstash", > > "vhost" : "logstash", > > "exchange" : "parsed\_logs", > > "exchange\_type" : "fanout", > > "exchange\_durable": "true", > > "queue" : "elasticsearch", > > "queue\_durable": "true", > > "routing\_key" : "elasticsearch" > > }, > > "index" : { > > "bulk\_size" : 100, > > "bulk\_timeout" : "10ms" > > }}' > > > > # netstat --tcp -ep | grep java | grep amqp ## ...but hey, it's still > > > > gone... > > $ curl -v -XPUT 'localhost:9200/\_river/logstash\_events\_201102220702/ > > \_meta' -d ' > > { > > "type" : "rabbitmq", > > "rabbitmq" : { > > "host" : "192.168.123.8", > > "port" : 5672, > > "user" : "logstash", > > "pass" : "logstash", > > "vhost" : "logstash", > > "exchange" : "parsed\_logs", > > "exchange\_type" : "fanout", > > "exchange\_durable": "true", > > "queue" : "elasticsearch", > > "queue\_durable": "true", > > "routing\_key" : "elasticsearch" > > }, > > "index" : { > > "bulk\_size" : 100, > > "bulk\_timeout" : "10ms" > > }}' > > > > # netstat --tcp -ep | grep java | grep amqp > > > > tcp 0 0 logs.vguest:56203 > > mon1.vguest:amqp ESTABLISHED elasticsearch 148361 21250/ > > java > > > > ...until we created it under a new name, at which time it works fine. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:11am UTC](https://discuss.elastic.co/t/river-not-being-migrated-to-new-node-on-service-restart-es-0-15-0/3959/5 "2017-07-06T04:11:42Z")