Using river for cyclical operations on the cluster


(Rafał Kuć) #1

Hello,

We developed a custom river plugin, to make some cyclical operations
on the cluster. What I'm worried about is that rivers are designed for
data indexation or at least that's what the Elasticsearch
documentation says about them. My question is if the approach we took
is a good one or should we change it ? One more thing - we need our
plugin to be singleton inside a cluster (just like river is), not to
do the same things again and again

Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch


(Shay Banon) #2

Can you explain a bit more what it does? You mean cyclical operations on the cluster as in repeating operations, like gathering stats? If so, then I would implement it differently. Node level stats can be reported for each node as a simple standalone (node level) service, and index level stats can be gathered from the master node (you can tell if the node is master or not by registering for cluster events).

On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote:

Hello,

We developed a custom river plugin, to make some cyclical operations
on the cluster. What I'm worried about is that rivers are designed for
data indexation or at least that's what the Elasticsearch
documentation says about them. My question is if the approach we took
is a good one or should we change it ? One more thing - we need our
plugin to be singleton inside a cluster (just like river is), not to
do the same things again and again

Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch


(Rafał Kuć) #3

Hi,

It's rather a cyclical index creation with setting tags for nodes,
changing filtering and stuff like that. It's not about monitoring.

Regards,
Rafał

On 9 Lut, 09:25, Shay Banon kim...@gmail.com wrote:

Can you explain a bit more what it does? You mean cyclical operations on the cluster as in repeating operations, like gathering stats? If so, then I would implement it differently. Node level stats can be reported for each node as a simple standalone (node level) service, and index level stats can be gathered from the master node (you can tell if the node is master or not by registering for cluster events).

On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote:

Hello,

We developed a custom river plugin, to make some cyclical operations
on the cluster. What I'm worried about is that rivers are designed for
data indexation or at least that's what the Elasticsearch
documentation says about them. My question is if the approach we took
is a good one or should we change it ? One more thing - we need our
plugin to be singleton inside a cluster (just like river is), not to
do the same things again and again

Regards,
Rafał Kuć
Sematext ::http://sematext.com/:: Solr - Lucene - Nutch


(Otis Gospodnetić) #4

Hi Shay,

So as Rafal mentioned, this is not about monitoring.

Here's the situation:
We are building a massive ES cluster (hundreds of big servers,
possibly even over 1000 of them on day one). The volume of data is
huge and it just keeps coming. Thus, the cluster just needs to keep
expanding and expanding. We need to be able to search the last N days
(e.g. 30) worth of content, but also the whole index. Because of the
scale of things we want to make sure we maximize hardware utilization
by spreading everything veeeeeeery evenly.

The way to visualize this is as a grid where rows represent days 1-30
and columns represent shards and replicas.

For example:
Assuming a single server can hold half a day's worth of data, each day
we'll create a new index with 2 shards.
Each of these shards will live on a separate box.
In addition, we'll have N replicas, say a replica for each of those 2
shards.
You can visualize the above as a row of 4 servers: 2 servers for 2
shards, and 1 server for a replica of each of the 2 shards - 4 servers
total.

So on day 1 we'll have a row of 4 boxes.
On day 2 another row of 4 boxes.
and so on...

Because data is coming in continuously, we thought it may be best to
have a separate "process" (separate from the indexer) that pre-creates
indices on specific machines (using shard/replica include/exclude
allocation tags). For example, some time on Monday it would pre-
create an index on the row of boxes that we want holding Tuesday's
index. This really could be a completely standalone/separate process
running on one of the ES nodes or externally, but then we have SPOF,
so we thought we'd implement this as a River, so that ES can manage it
and thus eliminate this SPOF.

All Rivers I see on ES site are for indexing content.
So I think the question is whether Rivers are purely meant for
indexing or if they can be used for anything one wants to run within
ES?

Thanks,
Otis

On Feb 9, 3:25 am, Shay Banon kim...@gmail.com wrote:

Can you explain a bit more what it does? You mean cyclical operations on the cluster as in repeating operations, like gathering stats? If so, then I would implement it differently. Node level stats can be reported for each node as a simple standalone (node level) service, and index level stats can be gathered from the master node (you can tell if the node is master or not by registering for cluster events).

On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote:

Hello,

We developed a custom river plugin, to make some cyclical operations
on the cluster. What I'm worried about is that rivers are designed for
data indexation or at least that's what the Elasticsearch
documentation says about them. My question is if the approach we took
is a good one or should we change it ? One more thing - we need our
plugin to be singleton inside a cluster (just like river is), not to
do the same things again and again

Regards,
Rafał Kuć
Sematext ::http://sematext.com/:: Solr - Lucene - Nutch


(Shay Banon) #5

Hi,

Yes, rivers can definitely be used for this purpose. Think of them as a generic process that can run in the cluster. Another option to handle it so to have the process tied to the master node in the cluster, for this, you can listen to a cluster event and have the process start when the node becomes a master node. But, rivers would do the trick as well.

On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote:

Hello,

We developed a custom river plugin, to make some cyclical operations
on the cluster. What I'm worried about is that rivers are designed for
data indexation or at least that's what the Elasticsearch
documentation says about them. My question is if the approach we took
is a good one or should we change it ? One more thing - we need our
plugin to be singleton inside a cluster (just like river is), not to
do the same things again and again

Regards,
Rafał Kuć
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch


(Rafał Kuć) #6

Thanks for the answer Shay.

Regards,
Rafał Kuć
Sematext ::http://sematext.com/:: Solr - Lucene - Nutch


(system) #7