# Custom TopDocsCollector in plugin?

**URL:** <https://discuss.elastic.co/t/custom-topdocscollector-in-plugin/286546>\
**Category:** Elasticsearch\
**Created:** [October 12, 2021, 8:19pm UTC](https://discuss.elastic.co/t/custom-topdocscollector-in-plugin/286546 "2021-10-12T20:19:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![larry\_benchling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larry_benchling/32/95773_2.png) [@larry\_benchling](https://discuss.elastic.co/u/larry_benchling)\
**Post date:** [October 12, 2021, 8:19pm UTC](https://discuss.elastic.co/t/custom-topdocscollector-in-plugin/286546/1 "2021-10-12T20:19:40Z")

</div>

Hello!

I'm working on a plugin to add some cheminformatics features to our Elasticsearch cluster. Several of these involve steps where there is an inverted index based screening stage (which fits well as a Lucene query), followed by a more expensive algorithm to eliminate false positives.

The nature of these algorithms allows for utilization of the `setMinCompetitiveScore` API to skip these expensive steps when sorting by score, which allows us to keep the query tractable even for non-selective queries. However, we would also like to be able to sort by other fields, which creates a challenge out of the box due to the way Lucene sorting works - it requires collecting all of the hits to sort without potentially losing hits.

One potential solution I'm considering is creating a custom doc collector and moving the check for a false positive there, so that the psuedocode would look something like this:

```auto
get next doc id from screening step
check sort value of doc against priority queue; if it won't make it into the top K hits, skip it
else run the expensive false positive check and collect result if it passes

```

However, I'm not sure if there is a place to inject this doc collector via a plugin - it's not obvious to me if the plugin interface supports that. Any guidance on how I can accomplish this? As a last resort, I was considering creating a custom action plugin that exposes our own search API, but I'd rather not have to keep that up to date with the evolution of changes in the main search API, since we'll be composing this with other search capabilities.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [October 12, 2021, 8:38pm UTC](https://discuss.elastic.co/t/custom-topdocscollector-in-plugin/286546/2 "2021-10-12T20:38:36Z")

</div>

I don't believe you can do it now for the top level search. You can do it for aggs now, but that doesn't let you touch the main search collector. If you are just doing an agg on the matches anyway it'd work fine. Aggs like `terms` delays collection of the sub-aggs until we know what buckets are selected. If that's the kind of thing you are after you can look there.

If not, is there some way you could hack it into a custom Weight? It's quite normal for plugins to make custom queries.

---

<div class="post-metadata">

**Author:** ![larry\_benchling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larry_benchling/32/95773_2.png) [@larry\_benchling](https://discuss.elastic.co/u/larry_benchling)\
**Post date:** [October 12, 2021, 10:18pm UTC](https://discuss.elastic.co/t/custom-topdocscollector-in-plugin/286546/3 "2021-10-12T22:18:50Z")

</div>

We are already building custom weights and scorers for the queries, but the issue is that AFAIK those don't have the required information about the top K sort values when sorting by something other than score. Am I mistaken there?

Unfortunately this isn't for aggs, we're using it to display traditional search results for things like chemical substructure matches.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2021, 10:19pm UTC](https://discuss.elastic.co/t/custom-topdocscollector-in-plugin/286546/4 "2021-11-09T22:19:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.