is it possible to use elasticsearch for live river requests?
I want to use elasticsearch to collect data from multiple APIs - but only
if someone is really searching for it.
For example: User searches for a boat available between xmas and newyear -
now i want to use the river plugin to fetch the data from all the APIs and
merge it into my index
by live request, do you mean the situation that user hits elasticsearch
with query, and then waits for elasticsearch river to pull data from some
API (before responding back to user)?
is so then i dont think the river is the way to go here
if in other hand you can populate your index a little bit later, then you
can use river for that
you can implement custom river that will go through list of most popular
queries, run them through API, and index
Cheer,
On Monday, December 23, 2013 6:13:30 PM UTC, chh wrote:
Hello,
is it possible to use elasticsearch for live river requests?
I want to use elasticsearch to collect data from multiple APIs - but only
if someone is really searching for it.
For example: User searches for a boat available between xmas and newyear -
now i want to use the river plugin to fetch the data from all the APIs and
merge it into my index
rivers are used to pull data in - but totally independent from incoming
search requests. You could write your own plugin, which intercepts a search
request and forwards it to your API (and indexes the results afterwards),
but it might make more sense to actually execute kind of 'search' in
another component of your architecture.
So, execute the query, return the results to the requester and store them
in your index, so you can use it for searches from then on.
Wondering about the use-case here, interested to know more.
--Alex
On Mon, Dec 23, 2013 at 7:13 PM, chh chh@miet24.de wrote:
Hello,
is it possible to use elasticsearch for live river requests?
I want to use elasticsearch to collect data from multiple APIs - but only
if someone is really searching for it.
For example: User searches for a boat available between xmas and newyear -
now i want to use the river plugin to fetch the data from all the APIs and
merge it into my index
It looks like your use case is federated search, the approach that let
users search for what they want, but in the background a dispatcher is
analyzing the request to forward the search to only relevant sources that
might be able to match the request.
Rivers are not the instrument for federated search. And there is no
solution for that, if you still want performance - the idea is to aggregate
all sources and consolidate it into a single ES cluster for the fastest
possible search.
Of course you can experiment by federating several ES clusters but in that
case you must do the heavy lifting (distributed indexing, ranking,
faceting, filtering) by your own code.
thanks for your answers. I hope everyone had a nice xmas.
Like Jörg posted: 'federated search' is what i was trying to accomplish
within ES - yet i forgot the name.
Would've been nice to have a single place to collect and serve the data.
Unfortunately there are too many options to simply index all the foreign
data.
Christian
Am Mittwoch, 25. Dezember 2013 17:24:18 UTC+1 schrieb Jörg Prante:
It looks like your use case is federated search, the approach that let
users search for what they want, but in the background a dispatcher is
analyzing the request to forward the search to only relevant sources that
might be able to match the request.
Rivers are not the instrument for federated search. And there is no
solution for that, if you still want performance - the idea is to aggregate
all sources and consolidate it into a single ES cluster for the fastest
possible search.
Of course you can experiment by federating several ES clusters but in that
case you must do the heavy lifting (distributed indexing, ranking,
faceting, filtering) by your own code.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.