Temporary disable river allocation (dynamic cluster setting?)

Elastic Stack Elasticsearch
1

Is it possible to temporary disable river allocation?

We want to separate the nodes running the rivers from the nodes
holding the data, so that the nodes running the rivers can be
temporary shut down or restarted without having an impact on the data
nodes. After restarting the river nodes, all our rivers get allocated
to the first node that joins the cluster (as it is the only node with
river set to true).

I suppose that a ClusterChangedEvent gets fired for one node after the
other, and that the RiversRouter assigns all the rivers after the
first node joining. I could not find an option to temporary disable
rivers allocation, because then I could wait till all the nodes have
been restarted and only then reenable river allocation. Or maybe add
an option to delay river allocation.

A working (but not very elegant solution) is to restart the river
nodes, then restart the single node holding all the rivers, which
causes the RiversRouter to redistribute the rivers uniformly among the
other nodes. (Seems somewhat hacky to me :wink: and has the disadvantage
that the double-restarted node has no river running)

Thanks in advance for any advice or comments,

Mechel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH0sEYifq8B3YOSmp7pk2yF%2B-XX-2yJwPu_MgXYNJNUvAUXZqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Maybe I do not understand fully, but you say "as it is the only node with
river set to true" - this means to me there is only this one node that
should run river instances - and then you have observed the RiversRouter
redistributes "uniformly" rivers to other nodes? I'm a bit confused where
you want to see river instances running.

The current river concept is that a river instance is running as a
singleton on the node where it has been started, and in case of a node
failure (or shutdown), another node takes over. There are several
challenges with that concept.

So I'm afraid that disabling river allocation will add more challenges to
the automatic failover idea.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%2BYR0sNgCv-%3Dx7WbzSjasPGuonzP_%2BZ1srsKh49wR6pA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi, maybe its clearer with an example.

Imagine I have 5 data node, 5 river nodes and 10 rivers running on
these 5 river nodes.
The data nodes are called N1 to N5, these have river set to false.
The river nodes are called R1 to R5, these have river set to true.
After creating the rivers or after a full cluster restart everything
is nicely balanced, each of the 5 river nodes has exactly 2 rivers.

I then want to restart the river nodes R1-R5 without restarting N1-N5.
The problem here is that the first node that comes up again and joins
the cluster gets all the 10 rivers assigned.

I have no issue with having a small imbalance between the number of
rivers running per river node, but I want to prevent that all the
rivers run on one node,
therefore I wanted to turn off or throttle river allocation before
doing the partial cluster restart and turing it back on, when all (or
most) of the node are up again.

If that does not work, I could still use the 'hacky' solution, which does:
Restart R1-R5.
Node R1 (as an example) comes up first and gets all the 10 rivers.
Node R2-R5 come up and get no rivers (as R1 already has all of them)
Restart R1.
Then R2-R5 get the rivers distributed among them, which is acceptable
for my use-case, even if R1 has none.
But I somehow don't like the idea of restarting R1 twice just for
redistributing the rivers, so I was wondering if there is a better
solution, like adding a transient cluster setting which skips
allocation.

Thanks for any ideas,
Mechel

On Tue, Jan 14, 2014 at 4:31 PM, joergprante@gmail.com
joergprante@gmail.com wrote:

Maybe I do not understand fully, but you say "as it is the only node with
river set to true" - this means to me there is only this one node that
should run river instances - and then you have observed the RiversRouter
redistributes "uniformly" rivers to other nodes? I'm a bit confused where
you want to see river instances running.

The current river concept is that a river instance is running as a singleton
on the node where it has been started, and in case of a node failure (or
shutdown), another node takes over. There are several challenges with that
concept.

So I'm afraid that disabling river allocation will add more challenges to
the automatic failover idea.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%2BYR0sNgCv-%3Dx7WbzSjasPGuonzP_%2BZ1srsKh49wR6pA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAH0sEYhiqp0qx0VcEVwaqyegSxNv032cRsbuHMjPH16ogrcVUw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I think I understand your scenario, at least I hope I made some progress...

The challenge is to coordinate river execution on independent nodes,
especially during a critical cluster restart phase. Because river instances
are autonomous by design, the result is unbalanced - the ES RiversRouter
does not care how many river instances are "allowed" on a node, so it
happily starts 10 rivers on a node that were previously on 5 nodes.

To get the affinity problem addressed, I hope I can make progress on a
"gatherer" framework that is able to distribute jobs among many nodes:

Each gatherer executes jobs by a thread pool. To avoid overloading jobs on
a single gatherer node (e.g. in a critical cluster restart phase) I plan to
let a reschedule thread check the cluster state for available gatherer
nodes and their capacities (which can be the job queue length, or the
system load). Jobs, once started, are owned by gatherer nodes, and added to
a waiting queue. The queue can be examined, and jobs could also be
reassigned to another gatherer with more available capacity. That is the
basic idea.

Sorry, I know the gatherer project does not help quickly to improve the
current river installations. Just my 2c to let you know about one of my
spare time activities :wink:

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE%2BA8pNV3EdYdPLtgW%3DjU0p7_2gVFaUpYZFXhdVB%2BBkqg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.