I am wondering if the percolate query in ES 5 (or in 2.3) allows full-text style similarity queries. If possible I would like to percolate existing and new documents and only 'tag/classify' them if the similarity score is above a certain threshold value. Is this possible out of the box, can it be done with a script, or not at all.
I understand that for this to work I would need to have access to the tf-idf of my index of documents to be percolated, which will become computationally intensive as the size of said index grows. But if I'm not mistaken percolation works differently in ES5 compared to ES2.3?
In ES 2.x and before scoring based on the document being percolated isn't possible.
In ES 5 many things have changed in the percolator and the percolate query does produce a score based on the document being percolated. The document being percolated is indexed into an temporary in-memory index as part of query parsing. This index only contains a single document and at search time all percolator queries are executed on that index. So when it comes to scoring for each percolator query, only the tf statistics has an impact, the df statistic is always 1. There is no way you can get access to the df statistic of another ES index.
The thresholding you mention can be achieved using min_score
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.