Hi, the main question is in the title. Elasticsearch version: 6.2
I want to say that I've already tried to find information for this task, and actually I've found some, but for one or another reason it doesn't work for documents containing massive strings in one text field.
As far as I get there are two most useful methods in finding duplicate docs:
- Using aggregations
I've read about this method here
Difficulties in using it:
- we can aggregate by text fields only with fielddata enabled, but in references it is not recommended to enable it because "Fielddata can consume a lot of heap space, especially when loading high cardinality text fields. Once fielddata has been loaded into the heap, it remains there for the lifetime of the segment. Also, loading fielddata is an expensive process which can cause users to experience latency hits." So is it really that bad? I guess it is for my index with like ~50000 docs (for now) with huge strings in one field (kibana freezes for some time to show even 1 doc)
- in theory we can aggregate by .raw keyword subfields, but as I've found here "Lucene doesn't allow terms that contain more than 32k bytes."
- More Like This Query
I've read about it here and of course in references for ES 6.2
Difficulties in using it:
- it can find duplicates only for one doc per request, not great, but ok
- it actually find similar not duplicate docs and I think it was made to do so, but can it be used to find exact same docs?
- I can't use keyword analyzer because Lucene doesn't allow terms that contain more than 32k bytes
- the main parameter in this query in this situation is minimum_should_match, which I tried to set to "100%", but for some reason it doesn't work, when I use "minimum_should_match": "100%", it works like it ignores it (I'm using Kibana for testing), actually in fact it works only with natural numbers like "minimum_should_match": 200, but not with negative integers and so on, like it is written here
This will work fine:
GET another_test_index/_search
{
"query": {
"more_like_this": {
"fields": ["text_field"],
"like": "I love oranges",
"min_term_freq" : 1,
"min_doc_freq": 1,
"max_query_terms" : 12,
"minimum_should_match": 3
}
}
}
but it won't work as expected with "minimum_should_match": "100%"
So without "minimum_should_match": "100%" it's kinda impossible to find duplicates using More Like This Query (keyword analyzer won't work);
even with "minimum_should_match": "100%", there is small detail - even if all terms are equal, it doesn't mean that we've found the exact same document, I guess it's not so big deal when working with really huge strings in one doc, but not nice
So:
- Are there any other methods which I can use in my situation?
- How bad can enabling fielddata for text fields here be?
- Why "minimum_should_match": "100%" doesn't work for More Like This Query or maybe I don't get something?
Any help or advice would be much appreciated!
