# Using river for cyclical operations on the cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [February 8, 2012, 8:51pm UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647 "2012-02-08T20:51:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [February 8, 2012, 8:51pm UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/1 "2012-02-08T20:51:56Z")
Hello, We developed a custom river plugin, to make some cyclical operations on the cluster. What I'm worried about is that rivers are designed for data indexation or at least that's what the Elasticsearch documentation says about them. My question is if the approach we took is a good one or should we change it ? One more thing - we need our plugin to be singleton inside a cluster (just like river is), not to do the same things again and again Regards, Rafał Kuć Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 9, 2012, 8:25am UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/2 "2012-02-09T08:25:29Z")
Can you explain a bit more what it does? You mean cyclical operations on the cluster as in repeating operations, like gathering stats? If so, then I would implement it differently. Node level stats can be reported for each node as a simple standalone (node level) service, and index level stats can be gathered from the master node (you can tell if the node is master or not by registering for cluster events). On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote: > Hello, > > We developed a custom river plugin, to make some cyclical operations > on the cluster. What I'm worried about is that rivers are designed for > data indexation or at least that's what the Elasticsearch > documentation says about them. My question is if the approach we took > is a good one or should we change it ? One more thing - we need our > plugin to be singleton inside a cluster (just like river is), not to > do the same things again and again > > Regards, > Rafał Kuć > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch ---
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [February 9, 2012, 8:36am UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/3 "2012-02-09T08:36:14Z")
Hi, It's rather a cyclical index creation with setting tags for nodes, changing filtering and stuff like that. It's not about monitoring. Regards, Rafał On 9 Lut, 09:25, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Can you explain a bit more what it does? You mean cyclical operations on the cluster as in repeating operations, like gathering stats? If so, then I would implement it differently. Node level stats can be reported for each node as a simple standalone (node level) service, and index level stats can be gathered from the master node (you can tell if the node is master or not by registering for cluster events). > > On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote: > > > Hello, > > > We developed a custom river plugin, to make some cyclical operations > > on the cluster. What I'm worried about is that rivers are designed for > > data indexation or at least that's what the Elasticsearch > > documentation says about them. My question is if the approach we took > > is a good one or should we change it ? One more thing - we need our > > plugin to be singleton inside a cluster (just like river is), not to > > do the same things again and again > > > Regards, > > Rafał Kuć > > Sematext ::[http://sematext.com/::](http://sematext.com/::) Solr - Lucene - Nutch ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [February 10, 2012, 4:10am UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/4 "2012-02-10T04:10:30Z")
Hi Shay, So as Rafal mentioned, this is not about monitoring. Here's the situation: We are building a massive ES cluster (hundreds of big servers, possibly even over 1000 of them on day one). The volume of data is huge and it just keeps coming. Thus, the cluster just needs to keep expanding and expanding. We need to be able to search the last N days (e.g. 30) worth of content, but also the whole index. Because of the scale of things we want to make sure we maximize hardware utilization by spreading everything veeeeeeery evenly. The way to visualize this is as a grid where rows represent days 1-30 and columns represent shards and replicas. For example: Assuming a single server can hold half a day's worth of data, each day we'll create a new index with 2 shards. Each of these shards will live on a separate box. In addition, we'll have N replicas, say a replica for each of those 2 shards. You can visualize the above as a row of 4 servers: 2 servers for 2 shards, and 1 server for a replica of each of the 2 shards - 4 servers total. So on day 1 we'll have a row of 4 boxes. On day 2 another row of 4 boxes. and so on... Because data is coming in continuously, we thought it may be best to have a separate "process" (separate from the indexer) that pre-creates indices on specific machines (using shard/replica include/exclude allocation tags). For example, some time on Monday it would pre- create an index on the row of boxes that we want holding Tuesday's index. This really could be a completely standalone/separate process running on one of the ES nodes or externally, but then we have SPOF, so we thought we'd implement this as a River, so that ES can manage it and thus eliminate this SPOF. All Rivers I see on ES site are for indexing content. So I think the question is whether Rivers are purely meant for indexing or if they can be used for anything one wants to run within ES? ## Thanks, Otis > **[Sematext | IT System Monitoring Tools for DevOps](https://sematext.com/)** > > IT system monitoring and management tools for DevOps who need 24x7 live visibility into their infrastructure. On Feb 9, 3:25 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Can you explain a bit more what it does? You mean cyclical operations on the cluster as in repeating operations, like gathering stats? If so, then I would implement it differently. Node level stats can be reported for each node as a simple standalone (node level) service, and index level stats can be gathered from the master node (you can tell if the node is master or not by registering for cluster events). > > On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote: > > > Hello, > > > We developed a custom river plugin, to make some cyclical operations > > on the cluster. What I'm worried about is that rivers are designed for > > data indexation or at least that's what the Elasticsearch > > documentation says about them. My question is if the approach we took > > is a good one or should we change it ? One more thing - we need our > > plugin to be singleton inside a cluster (just like river is), not to > > do the same things again and again > > > Regards, > > Rafał Kuć > > Sematext ::[http://sematext.com/::](http://sematext.com/::) Solr - Lucene - Nutch ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 12, 2012, 1:15pm UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/5 "2012-02-12T13:15:22Z")
Hi, Yes, rivers can definitely be used for this purpose. Think of them as a generic process that can run in the cluster. Another option to handle it so to have the process tied to the master node in the cluster, for this, you can listen to a cluster event and have the process start when the node becomes a master node. But, rivers would do the trick as well. On Wednesday, February 8, 2012 at 10:51 PM, Rafał Kuć wrote: > Hello, > > We developed a custom river plugin, to make some cyclical operations > on the cluster. What I'm worried about is that rivers are designed for > data indexation or at least that's what the Elasticsearch > documentation says about them. My question is if the approach we took > is a good one or should we change it ? One more thing - we need our > plugin to be singleton inside a cluster (just like river is), not to > do the same things again and again > > Regards, > Rafał Kuć > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch ---
**Author:** ![Rafal\_Kuc\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rafal_kuc_3/32/799_2.png) [@Rafal\_Kuc\_3](https://discuss.elastic.co/u/Rafal_Kuc_3)\ **Post date:** [February 12, 2012, 6:02pm UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/6 "2012-02-12T18:02:24Z")
Thanks for the answer Shay. Regards, Rafał Kuć Sematext ::[http://sematext.com/::](http://sematext.com/::) Solr - Lucene - Nutch ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:39am UTC](https://discuss.elastic.co/t/using-river-for-cyclical-operations-on-the-cluster/6647/7 "2017-07-06T03:39:41Z")