Dedup Records According to Percentage

[Dawood_Siddiq]

Greetings : I have Above than 1M records in ES. Now i want to dedup data on bases of percentage. For Example "Give me list of all records whose title are 90% matched or above".
Lets take another example. "I need to retrieve all records whose locations are 80% or above matched".

I try to dedup records according to title but i need to retrieve by percentage.

GET index/_search
{
  "size": 0, 
  "aggs": {
    "duplicate": {
         "terms": {
              "script": "doc['product_title'].value",
              "size": 1000
             },
         "aggs": {
            "duplicate list": {
            "top_hits": {
            }
        }
      }
    }
  } 
}

How can i fetch duplicate records whose specific column match by defined percentage with other records. Any help would be appreciable. Thanks

[xeraa]

What is a 90% match of a title? 90% of the characters are the same?

Generally, if you try to express scores in percentage it won't end well and the Lucene docs are pretty explicit about that.

[Dawood_Siddiq]

thanks for reply @xeraa . what else i do to find solution of this kind of problem ?

[xeraa]

For starters, what is an 80% match? How do you calculate that with concrete examples?

[Dawood_Siddiq]

For example i have title "Greenland swimming Pools Georgia" ...
Now 100% match is definitely we all know . if i have a record having title "swimming pools georgia" then i think its similar to previous one approx 70% . I just want to dig out the algorithm who can help me to fetch these similar records with scoring .

Is any way to find similar records like this in ES ?

[xeraa]

Sounds like the more like this feature. Does that work for you?

[Dawood_Siddiq]

No @xeraa . more like this feature doesn't work for me .

[xeraa]

Ok, why not? We're not making much progress with that answer alone.