A question about the confluence of rivers

Elastic Stack Elasticsearch
1

my understanding is that when you define/create a river (such as the
rabbitmq river), you define an index it flows into.
my questions are:

  • can two rivers flow into one index?
  • can a river flow into an existing index?
  • the implied question is whether the same index can be fed both by a
    river and a normal indexing api call

and since we are on the subject of rivers, is there a hook for a river
to preprocess an entry before it is being index (river ->
[process-or-filter] -> index)

2

On Thu, Apr 12, 2012 at 11:07 AM, Benny Sadeh benny.sadeh@gmail.com wrote:

my understanding is that when you define/create a river (such as the
rabbitmq river), you define an index it flows into.
my questions are:

  • can two rivers flow into one index?

Yes, they simply perform index operations against that index.

  • can a river flow into an existing index?

Yep.

  • the implied question is whether the same index can be fed both by a
    river and a normal indexing api call

Yep.

and since we are on the subject of rivers, is there a hook for a river
to preprocess an entry before it is being index (river ->
[process-or-filter] -> index)

There is some support for it in certain rivers, the rabbitmq does not as it
assumes the message that you send is already processed.

3

thanks for answering Shay; you're always very responsive, and I
appreciate that :slight_smile:

On Apr 13, 5:16 am, Shay Banon kim...@gmail.com wrote:

On Thu, Apr 12, 2012 at 11:07 AM, Benny Sadeh benny.sa...@gmail.com wrote:

my understanding is that when you define/create a river (such as the
rabbitmq river), you define an index it flows into.
my questions are:

  • can two rivers flow into one index?

Yes, they simply perform index operations against that index.

  • can a river flow into an existing index?

Yep.

  • the implied question is whether the same index can be fed both by a
    river and a normal indexing api call

Yep.

great, but how?

for example, assuming I created an index such as:
curl -XPUT http://localhost:9200// -d '{ ...

then when creating a rabbitmq river, if I would specify:
curl -XPUT 'localhost:9200/_river//_meta' -d
'{
"type" : "rabbitmq",
"rabbitmq" : {
...
},
"index" : {
...
}
}'
I was assuming this would be a different index: _river/ rather the
am I wrong and this would be one and the same index?

and since we are on the subject of rivers, is there a hook for a river
to preprocess an entry before it is being index (river ->
[process-or-filter] -> index)

There is some support for it in certain rivers, the rabbitmq does not as it
assumes the message that you send is already processed.

so I assume if one wanted such [process-or-filter] river
functionality, one could just fork the river implementation and change
it to do so. I was just wondering if there is a "hook" method to do
so, or an agreed/by-convention way.
since I haven't looked into how to implement a river docs yet, if the
answer is in there, please tell me if I should just rtfm :wink: