Hooking into search request execution

Elastic Stack Elasticsearch
1

Hi,

In a different thread Shay wrote:

Elasticsearch execution is a bit different. Because all is segment based and collector based,
there are not different "stages" of search, all is done while the query executes.

So is there no way for one to "hook into" the search request
execution?
For example, right now we are working on a project with Solr and
building a SearchComponent that takes the query, analyzes it in a
certain way, takes some data out of it, executes some Java code to
transform the extracted data into something else and essentially
rewrites the query a bit. Implementation details are irrelevant - the
key is that I can write some custom Java code that extends Solr's
SearchComponent, declare this new component of mine in solrconfig.xml
and include it in the request flow. In this case, I'd put it before
the component that does the actually querying, but we've also built
components that came after the query component in this component
chain.

This was discussed in
http://groups.google.com/group/elasticsearch/browse_thread/thread/d0dbd5ea795fa8a4/5dfa56829a94ff60

...but there was no clear answer... and I'd love to know if Shay has
any plans around this for ES. I, obviously, find hitting this
repeatedly and I'm probably not alone.

Thanks,
Otis

Sematext is hiring world-wide-- Jobs - Sematext

2

First, this has nothing to do with my comment on the other thread.

For your use case, you can plug your own query parser into elasticsearch so
it will be part of the query DSL, and then do whatever you want there.
which at the end generates a Query. There are other specific points where
you want to plug your own custom specific logic. Another option is to
actually expose your own endpoint and slice and dice the search request
however you want (the solr protocol plugin that was written by the
community on github is a good example).

But, as a design decision, I personally don't believe that such logic
belong in elasticsearch (on a project level, not a generic plugin). You are
abusing it to try and put code there where it actually belongs in your app
layer. Its much simpler to manage code as part of your app layer than
having to manage code that is running in elasticsearch. Obviously, every
use case has its exception, but be weary that now you introduced another
dependency which is not elasticsearch API, but elasticsearch codebase,
instead of doing it yourself.

-shay.banon

On Sat, Jan 14, 2012 at 10:34 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

In a different thread Shay wrote:

Elasticsearch execution is a bit different. Because all is segment based
and collector based,
there are not different "stages" of search, all is done while the query
executes.

So is there no way for one to "hook into" the search request
execution?
For example, right now we are working on a project with Solr and
building a SearchComponent that takes the query, analyzes it in a
certain way, takes some data out of it, executes some Java code to
transform the extracted data into something else and essentially
rewrites the query a bit. Implementation details are irrelevant - the
key is that I can write some custom Java code that extends Solr's
SearchComponent, declare this new component of mine in solrconfig.xml
and include it in the request flow. In this case, I'd put it before
the component that does the actually querying, but we've also built
components that came after the query component in this component
chain.

This was discussed in

http://groups.google.com/group/elasticsearch/browse_thread/thread/d0dbd5ea795fa8a4/5dfa56829a94ff60

...but there was no clear answer... and I'd love to know if Shay has
any plans around this for ES. I, obviously, find hitting this
repeatedly and I'm probably not alone.

Thanks,
Otis

Sematext is hiring world-wide-- Jobs - Sematext