Using river plugin for live requests


(chh) #1

Hello,

is it possible to use elasticsearch for live river requests?
I want to use elasticsearch to collect data from multiple APIs - but only
if someone is really searching for it.

For example: User searches for a boat available between xmas and newyear -
now i want to use the river plugin to fetch the data from all the APIs and
merge it into my index

Best regards

Christian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db3b0a57-9261-4a31-8783-94edbbadbd3b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Karol Gwaj) #2

by live request, do you mean the situation that user hits elasticsearch
with query, and then waits for elasticsearch river to pull data from some
API (before responding back to user)?
is so then i dont think the river is the way to go here

if in other hand you can populate your index a little bit later, then you
can use river for that
you can implement custom river that will go through list of most popular
queries, run them through API, and index

Cheer,

On Monday, December 23, 2013 6:13:30 PM UTC, chh wrote:

Hello,

is it possible to use elasticsearch for live river requests?
I want to use elasticsearch to collect data from multiple APIs - but only
if someone is really searching for it.

For example: User searches for a boat available between xmas and newyear -
now i want to use the river plugin to fetch the data from all the APIs and
merge it into my index

Best regards

Christian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63d5c4d7-760c-47d7-b039-423838cee4e6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #3

Hey,

rivers are used to pull data in - but totally independent from incoming
search requests. You could write your own plugin, which intercepts a search
request and forwards it to your API (and indexes the results afterwards),
but it might make more sense to actually execute kind of 'search' in
another component of your architecture.

So, execute the query, return the results to the requester and store them
in your index, so you can use it for searches from then on.

Wondering about the use-case here, interested to know more.

--Alex

On Mon, Dec 23, 2013 at 7:13 PM, chh chh@miet24.de wrote:

Hello,

is it possible to use elasticsearch for live river requests?
I want to use elasticsearch to collect data from multiple APIs - but only
if someone is really searching for it.

For example: User searches for a boat available between xmas and newyear -
now i want to use the river plugin to fetch the data from all the APIs and
merge it into my index

Best regards

Christian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/db3b0a57-9261-4a31-8783-94edbbadbd3b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_r9s0moTN7%3DD8ox11N%2BLjwX%2B8bPxuCKT9nn6YQpOK4bw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #4

It looks like your use case is federated search, the approach that let
users search for what they want, but in the background a dispatcher is
analyzing the request to forward the search to only relevant sources that
might be able to match the request.

Rivers are not the instrument for federated search. And there is no
solution for that, if you still want performance - the idea is to aggregate
all sources and consolidate it into a single ES cluster for the fastest
possible search.

Of course you can experiment by federating several ES clusters but in that
case you must do the heavy lifting (distributed indexing, ranking,
faceting, filtering) by your own code.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF09oySa075vgsi1UShzMSj5h5T_atCN5hbAaZSDTtnsg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(chh) #5

Good morning,

thanks for your answers. I hope everyone had a nice xmas.

Like Jörg posted: 'federated search' is what i was trying to accomplish
within ES - yet i forgot the name.
Would've been nice to have a single place to collect and serve the data.
Unfortunately there are too many options to simply index all the foreign
data.

Christian

Am Mittwoch, 25. Dezember 2013 17:24:18 UTC+1 schrieb Jörg Prante:

It looks like your use case is federated search, the approach that let
users search for what they want, but in the background a dispatcher is
analyzing the request to forward the search to only relevant sources that
might be able to match the request.

Rivers are not the instrument for federated search. And there is no
solution for that, if you still want performance - the idea is to aggregate
all sources and consolidate it into a single ES cluster for the fastest
possible search.

Of course you can experiment by federating several ES clusters but in that
case you must do the heavy lifting (distributed indexing, ranking,
faceting, filtering) by your own code.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c432d105-dd5a-445c-a19c-d16a958d2470%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6