ES plugin to expand search results

[thampton (Tom Hampton)]

Is it possible to build a plugin that can add additional records to a query result before the aggregation of the data happens?

I have data external to the index that identifies additional documents from the index that I would like included. These are documents that do not match the search criteria, but that I know are relevant based on other data/analysis.

So I would like to inject these related documents into the results before the aggregation happens.

The basic idea is to check and external data source, for each matching document if there are related documents and pull them into the result.

Thanks

[Ivan (Ivan Brusic)]

You could write a plugin that modifies the query, but I would opt to modify
the query on the client side instead.

What does the external source provide? Additional document ids or
additional search criteria? For the former, simply add an Ids query [1]
alongside your original query in a should clause [2]

{
"query": {
"bool" : {
"should" : [
{ original query },
"ids" : {
"type" : "my_type",
"values" : ["1", "4", "100"]
}
]
}
}
}

I believe Solr supports this type of query, but nothing native to
Elasticsearch.

[1]
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-ids-query.html
[2]
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html

[thampton (Tom Hampton)]

@Ivan Thanks for your response.

The external data source provides additional document IDs, based on the documents returned from the query. So I don't have the extra document IDs until after I run the query.

1. run query to get keyword matches
2. for each matching document lookup related non-matching documents & add to hits
3. aggregate result
4. visualize in Kibana

Step 2 is a proprietary algorithm that works on a different data structure.

I'm hoping that an ES plugin can make steps 1,2,3 transparent to the client Kibana.