Can an Elasticsearch Search plugin filter results post-search?

[Dor_Levi]

Hi,

I'm trying to figure out a way to have the default search done by ES to filter out some of the hits after getting back from the core engine, before passing them as a result to the rest caller.

The actual filtering might be done by cross checking some file/config/different DB/etc..., and therefor can't (realistically) be done as part of the original query, only after actually getting the hits.

I found a way I can augment hits, by creating a Plugin that implements SearchPlugin and provides a FetchSubPhase that can modify the hits, but I didn't find any way to remove a hit altogether.

Is there a way to accomplish this?

Thanks

[spinscale (Alexander Reelsen)]

I don't think that there is good way to do this at query time within the Search instead of post processing.

Any chance you could enrich your data at index time, i.e. using the enrich processor?

If you tell a little more about your use-case, maybe there is a way to rethink the data structures/queries being used.

[Dor_Levi]

I cannot enrich my data in index time, as the information I'm filtering by comes from an external DB which can change dynamically.

So far the "best" solution I came up with is to have my ActionPlugin provide an ActionFilter, which doesn't really filter anything, but instead wraps the provided ActionListener with one of my own, allowing me to exclude any unwanted results from the response before it's being returned to the user:

public class MySearchFilter implements ActionFilter
{
	@Override
	public int order()
	{
		return Integer.MAX_VALUE;
	}
	
	@Override
	public <Request extends ActionRequest, Response extends ActionResponse> void apply(
			Task task, String action, Request request,
			ActionListener<Response> listener, ActionFilterChain<Request, Response> chain)
	{
		if (!(request instanceof SearchRequest))
		{
			chain.proceed(task, action, request, listener);
			return;
		}
		
		chain.proceed(task, action, request, new ActionListener<Response>()
		{
			@Override
			public void onResponse(Response response)
			{
				// DO WORK HERE
				
				listener.onResponse(response);
			}
			
			@Override
			public void onFailure(Exception e)
			{
				listener.onFailure(e);
			}
		});
	}
}

It works, but I'm wondering if there's a better way to achieve this?

[spinscale (Alexander Reelsen)]

While it is possible to put in there, let me add one final word of warning:

Unless you are fully aware of how threading works, and what threads you are blocking with the above code and how that may drastically reduce your query speed or even cluster stability due to blocking on certain threads, I highly encourage you do this kind of enrichment outside of Elasticsearch.

Also, you will need to upgrade your plugin with every Elasticsearch release, even including minors - which of course can be automated.

[Dor_Levi]

Thanks for the important pointers, I'll be sure to exercise caution and validate overall cluster behavior and stability.