# Temporary disable river allocation (dynamic cluster setting?) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 14, 2014, 1:53pm UTC](https://discuss.elastic.co/t/temporary-disable-river-allocation-dynamic-cluster-setting/15233 "2014-01-14T13:53:46Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [January 14, 2014, 1:53pm UTC](https://discuss.elastic.co/t/temporary-disable-river-allocation-dynamic-cluster-setting/15233/1 "2014-01-14T13:53:46Z")
Is it possible to temporary disable river allocation? We want to separate the nodes running the rivers from the nodes holding the data, so that the nodes running the rivers can be temporary shut down or restarted without having an impact on the data nodes. After restarting the river nodes, all our rivers get allocated to the first node that joins the cluster (as it is the only node with river set to true). I suppose that a ClusterChangedEvent gets fired for one node after the other, and that the RiversRouter assigns all the rivers after the first node joining. I could not find an option to temporary disable rivers allocation, because then I could wait till all the nodes have been restarted and only then reenable river allocation. Or maybe add an option to delay river allocation. A working (but not very elegant solution) is to restart the river nodes, then restart the single node holding all the rivers, which causes the RiversRouter to redistribute the rivers uniformly among the other nodes. (Seems somewhat hacky to me 😉 and has the disadvantage that the double-restarted node has no river running) Thanks in advance for any advice or comments, Mechel -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH0sEYifq8B3YOSmp7pk2yF%2B-XX-2yJwPu\_MgXYNJNUvAUXZqQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH0sEYifq8B3YOSmp7pk2yF%2B-XX-2yJwPu_MgXYNJNUvAUXZqQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 14, 2014, 3:31pm UTC](https://discuss.elastic.co/t/temporary-disable-river-allocation-dynamic-cluster-setting/15233/2 "2014-01-14T15:31:31Z")
Maybe I do not understand fully, but you say "as it is the only node with river set to true" - this means to me there is only this one node that should run river instances - and then you have observed the RiversRouter redistributes "uniformly" rivers to other nodes? I'm a bit confused where you want to see river instances running. The current river concept is that a river instance is running as a singleton on the node where it has been started, and in case of a node failure (or shutdown), another node takes over. There are several challenges with that concept. So I'm afraid that disabling river allocation will add more challenges to the automatic failover idea. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%2BYR0sNgCv-%3Dx7WbzSjasPGuonzP\_%2BZ1srsKh49wR6pA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%2BYR0sNgCv-%3Dx7WbzSjasPGuonzP_%2BZ1srsKh49wR6pA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Michel\_Conrad](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michel\_Conrad](https://discuss.elastic.co/u/Michel_Conrad)\ **Post date:** [January 14, 2014, 4:08pm UTC](https://discuss.elastic.co/t/temporary-disable-river-allocation-dynamic-cluster-setting/15233/3 "2014-01-14T16:08:28Z")
Hi, maybe its clearer with an example. Imagine I have 5 data node, 5 river nodes and 10 rivers running on these 5 river nodes. The data nodes are called N1 to N5, these have river set to false. The river nodes are called R1 to R5, these have river set to true. After creating the rivers or after a full cluster restart everything is nicely balanced, each of the 5 river nodes has exactly 2 rivers. I then want to restart the river nodes R1-R5 without restarting N1-N5. The problem here is that the first node that comes up again and joins the cluster gets all the 10 rivers assigned. I have no issue with having a small imbalance between the number of rivers running per river node, but I want to prevent that all the rivers run on one node, therefore I wanted to turn off or throttle river allocation before doing the partial cluster restart and turing it back on, when all (or most) of the node are up again. If that does not work, I could still use the 'hacky' solution, which does: Restart R1-R5. Node R1 (as an example) comes up first and gets all the 10 rivers. Node R2-R5 come up and get no rivers (as R1 already has all of them) Restart R1. Then R2-R5 get the rivers distributed among them, which is acceptable for my use-case, even if R1 has none. But I somehow don't like the idea of restarting R1 twice just for redistributing the rivers, so I was wondering if there is a better solution, like adding a transient cluster setting which skips allocation. Thanks for any ideas, Mechel On Tue, Jan 14, 2014 at 4:31 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Maybe I do not understand fully, but you say "as it is the only node with > river set to true" - this means to me there is only this one node that > should run river instances - and then you have observed the RiversRouter > redistributes "uniformly" rivers to other nodes? I'm a bit confused where > you want to see river instances running. > > The current river concept is that a river instance is running as a singleton > on the node where it has been started, and in case of a node failure (or > shutdown), another node takes over. There are several challenges with that > concept. > > So I'm afraid that disabling river allocation will add more challenges to > the automatic failover idea. > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%2BYR0sNgCv-%3Dx7WbzSjasPGuonzP\_%2BZ1srsKh49wR6pA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%2BYR0sNgCv-%3Dx7WbzSjasPGuonzP_%2BZ1srsKh49wR6pA%40mail.gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH0sEYhiqp0qx0VcEVwaqyegSxNv032cRsbuHMjPH16ogrcVUw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH0sEYhiqp0qx0VcEVwaqyegSxNv032cRsbuHMjPH16ogrcVUw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 14, 2014, 8:38pm UTC](https://discuss.elastic.co/t/temporary-disable-river-allocation-dynamic-cluster-setting/15233/4 "2014-01-14T20:38:18Z")
I think I understand your scenario, at least I hope I made some progress... The challenge is to coordinate river execution on independent nodes, especially during a critical cluster restart phase. Because river instances are autonomous by design, the result is unbalanced - the ES RiversRouter does not care how many river instances are "allowed" on a node, so it happily starts 10 rivers on a node that were previously on 5 nodes. To get the affinity problem addressed, I hope I can make progress on a "gatherer" framework that is able to distribute jobs among many nodes: > **[jprante/elasticsearch-gatherer](https://github.com/jprante/elasticsearch-gatherer)** > > elasticsearch-gatherer - Gatherers are scalable components for fetching and indexing data into Elasticsearch. They are designed as replacement for rivers. Each gatherer executes jobs by a thread pool. To avoid overloading jobs on a single gatherer node (e.g. in a critical cluster restart phase) I plan to let a reschedule thread check the cluster state for available gatherer nodes and their capacities (which can be the job queue length, or the system load). Jobs, once started, are owned by gatherer nodes, and added to a waiting queue. The queue can be examined, and jobs could also be reassigned to another gatherer with more available capacity. That is the basic idea. Sorry, I know the gatherer project does not help quickly to improve the current river installations. Just my 2c to let you know about one of my spare time activities 😉 Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE%2BA8pNV3EdYdPLtgW%3DjU0p7\_2gVFaUpYZFXhdVB%2BBkqg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE%2BA8pNV3EdYdPLtgW%3DjU0p7_2gVFaUpYZFXhdVB%2BBkqg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:56am UTC](https://discuss.elastic.co/t/temporary-disable-river-allocation-dynamic-cluster-setting/15233/5 "2017-07-06T01:56:48Z")
Closed on July 6, 2017, 1:56am UTC