Hi, I'm using the latest stable ElasticSearch (2.3.3) and I am using the More Like This query to find similar documents. My documents are highly structured with lots of fields. The problem I am running into occurs when large groups, or even most of my documents contain fields with the same terms (values): the More Like This query becomes slower and slower as more documents are added. Initially, I tried using the max_doc_freq to solve this problem. While max_doc_freq did solve the performance problem, it creates a new problem in that sometimes the More Like This query returns 0 results even when there are many similar documents in the index once the max_doc_freq threshold is exceeded for too many fields. I'm also aware of the stop words option, but I don't have advance knowledge of what words will become common.
It is my understanding that the performance impact comes from retrieving all the documents with any words in common (when max_doc_freq is not used) from the index and then computing scores for each one.
What I want is some way to run the More Like This query in such a way so that it maintains the performance while also returning similar documents. For example, if there were some way to only retrieve some documents (like 10 or 100) with common words that exceed the max_doc_freq, rather than every document, from the index, but still apply the full scoring function to every field regardless of document frequency. This way, if I inserted 1 million identical documents and set the max_doc_freq to 1000, I would still get some results rather than 0 when running More Like This for one of the 1 million identical documents, and for documents with less common terms, More Like This would work as expected.
Thanks!