River on Cluster

Elastic Stack Elasticsearch
1

Good Morning,

I have a single node elastic search cluster with two twitter rivers working
fine.

I am currently working on setting up a three node cluster to pull those
same two rivers. When we submit the rivers using curl we receive status
'ok'. However only the first river we add appears in the head plugin. It
does gather data successfully.

Can you use multiple rivers on a multi-node cluster? if yes, whats the
trick? :stuck_out_tongue_winking_eye:

Thank you for your time,

Arni

--

2

Hello!

Look here - http://www.elasticsearch.org/guide/reference/river/ - rivers are singletons inside a cluster, which means that only a single river of a given type will be allowed to work in a single cluster.

--

Regards,

Rafał Kuć

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch

Good Morning,

I have a single node elastic search cluster with two twitter rivers working fine.

I am currently working on setting up a three node cluster to pull those same two rivers. When we submit the rivers using curl we receive status 'ok'. However only the first river we add appears in the head plugin. It does gather data successfully.

Can you use multiple rivers on a multi-node cluster? if yes, whats the trick? :stuck_out_tongue_winking_eye:

Thank you for your time,

Arni

--

3

Hi Arni,

can you eventually provide the commands you used for setting up the rivers?
It should work. You can set up many rivers, but beware, they need different
names.

Jörg

On Monday, October 15, 2012 5:04:53 PM UTC+2, Arni Sumarlidason wrote:

Good Morning,

I have a single node Elasticsearch cluster with two twitter rivers
working fine.

I am currently working on setting up a three node cluster to pull those
same two rivers. When we submit the rivers using curl we receive status
'ok'. However only the first river we add appears in the head plugin. It
does gather data successfully.

Can you use multiple rivers on a multi-node cluster? if yes, whats the
trick? :stuck_out_tongue_winking_eye:

Thank you for your time,

Arni

--

4

please find attached,,

Thank you again for your time.

On Monday, October 15, 2012 11:13:28 AM UTC-4, Jörg Prante wrote:

Hi Arni,

can you eventually provide the commands you used for setting up the
rivers? It should work. You can set up many rivers, but beware, they need
different names.

Jörg

On Monday, October 15, 2012 5:04:53 PM UTC+2, Arni Sumarlidason wrote:

Good Morning,

I have a single node Elasticsearch cluster with two twitter rivers
working fine.

I am currently working on setting up a three node cluster to pull those
same two rivers. When we submit the rivers using curl we receive status
'ok'. However only the first river we add appears in the head plugin. It
does gather data successfully.

Can you use multiple rivers on a multi-node cluster? if yes, whats the
trick? :stuck_out_tongue_winking_eye:

Thank you for your time,

Arni

--

5

What it looks like to us:

  1. Even if we did setup two rivers covering different geospatial locations
    ~ the work load would not be distributed amongst the cluster. Meaning that
    the only benefit we would see from the cluster is in data redundancy and
    data retrieval speed.

It would have been nice to assign geospatial locations to specific
computers, but at this point our throughput isn't high enough to demand
such a setup. It looks like we will move to a two node system to ensure
up-time on the stream. That being said, there is a setting along the lines
of "how many nodes are required for the cluster to be considered
operational".. in the two node case, wouldn't 0 be the best answer here?
meaning a solo node is operational?

Thanks again for responding,

On Monday, October 15, 2012 11:26:10 AM UTC-4, Arni Sumarlidason wrote:

please find attached,,

Thank you again for your time.

On Monday, October 15, 2012 11:13:28 AM UTC-4, Jörg Prante wrote:

Hi Arni,

can you eventually provide the commands you used for setting up the
rivers? It should work. You can set up many rivers, but beware, they need
different names.

Jörg

On Monday, October 15, 2012 5:04:53 PM UTC+2, Arni Sumarlidason wrote:

Good Morning,

I have a single node Elasticsearch cluster with two twitter rivers
working fine.

I am currently working on setting up a three node cluster to pull those
same two rivers. When we submit the rivers using curl we receive status
'ok'. However only the first river we add appears in the head plugin. It
does gather data successfully.

Can you use multiple rivers on a multi-node cluster? if yes, whats the
trick? :stuck_out_tongue_winking_eye:

Thank you for your time,

Arni

--

6

You have started two rivers with a PUT command against localhost:9200, and
that is the same node, so don't expect a cluster moves rivers around the
nodes for workload distribution. The cluster just ensures that rivers are
running as steady singleton instances at a single node once installed.
Multi-node clusters scale for searching and indexing, which is triggered by
river instances, when they move fetched data into the cluster. Fetching
data is cheap compared to the processing within the cluster.

You need two nodes for a green cluster state, since it is the smallest
configuration that is safe against unexpected outage of one of the nodes.

Jörg

On Monday, October 15, 2012 7:20:52 PM UTC+2, Arni Sumarlidason wrote:

What it looks like to us:

  1. Even if we did setup two rivers covering different geospatial locations
    ~ the work load would not be distributed amongst the cluster. Meaning that
    the only benefit we would see from the cluster is in data redundancy and
    data retrieval speed.

It would have been nice to assign geospatial locations to specific
computers, but at this point our throughput isn't high enough to demand
such a setup. It looks like we will move to a two node system to ensure
up-time on the stream. That being said, there is a setting along the lines
of "how many nodes are required for the cluster to be considered
operational".. in the two node case, wouldn't 0 be the best answer here?
meaning a solo node is operational?

Thanks again for responding,

On Monday, October 15, 2012 11:26:10 AM UTC-4, Arni Sumarlidason wrote:

please find attached,,

Thank you again for your time.

On Monday, October 15, 2012 11:13:28 AM UTC-4, Jörg Prante wrote:

Hi Arni,

can you eventually provide the commands you used for setting up the
rivers? It should work. You can set up many rivers, but beware, they need
different names.

Jörg

On Monday, October 15, 2012 5:04:53 PM UTC+2, Arni Sumarlidason wrote:

Good Morning,

I have a single node Elasticsearch cluster with two twitter rivers
working fine.

I am currently working on setting up a three node cluster to pull those
same two rivers. When we submit the rivers using curl we receive status
'ok'. However only the first river we add appears in the head plugin. It
does gather data successfully.

Can you use multiple rivers on a multi-node cluster? if yes, whats the
trick? :stuck_out_tongue_winking_eye:

Thank you for your time,

Arni

--