# Dedup Records According to Percentage

**URL:** <https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025>\
**Category:** Elasticsearch\
**Created:** [August 6, 2019, 1:59pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025 "2019-08-06T13:59:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dawood\_Siddiq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawood_siddiq/32/51756_2.png) [@Dawood\_Siddiq](https://discuss.elastic.co/u/Dawood_Siddiq)\
**Post date:** [August 6, 2019, 1:59pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/1 "2019-08-06T13:59:55Z")

</div>

Greetings : I have Above than 1M records in ES. Now i want to dedup data on bases of percentage. For Example "Give me list of all records whose title are 90% matched or above".  
Lets take another example. "I need to retrieve all records whose locations are 80% or above matched".

I try to dedup records according to title but i need to retrieve by percentage.

```
GET index/_search
{
  "size": 0, 
  "aggs": {
    "duplicate": {
         "terms": {
              "script": "doc['product_title'].value",
              "size": 1000
             },
         "aggs": {
            "duplicate list": {
            "top_hits": {
            }
        }
      }
    }
  } 
}

```

How can i fetch duplicate records whose specific column match by defined percentage with other records. Any help would be appreciable. Thanks

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [August 6, 2019, 2:56pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/2 "2019-08-06T14:56:09Z")

</div>

What is a 90% match of a title? 90% of the characters are the same?

Generally, if you try to [express scores in percentage it won't end well and the Lucene docs are pretty explicit about that](https://cwiki.apache.org/confluence/display/LUCENEJAVA/ScoresAsPercentages).

---

<div class="post-metadata">

**Author:** ![Dawood\_Siddiq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawood_siddiq/32/51756_2.png) [@Dawood\_Siddiq](https://discuss.elastic.co/u/Dawood_Siddiq)\
**Post date:** [August 6, 2019, 3:23pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/3 "2019-08-06T15:23:34Z")

</div>

thanks for reply @xeraa . what else i do to find solution of this kind of problem ?

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [August 6, 2019, 9:42pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/4 "2019-08-06T21:42:49Z")

</div>

For starters, what is an 80% match? How do you calculate that with concrete examples?

---

<div class="post-metadata">

**Author:** ![Dawood\_Siddiq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawood_siddiq/32/51756_2.png) [@Dawood\_Siddiq](https://discuss.elastic.co/u/Dawood_Siddiq)\
**Post date:** [August 7, 2019, 9:21am UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/5 "2019-08-07T09:21:17Z")

</div>

For example i have title "Greenland swimming Pools Georgia" ...  
Now 100% match is definitely we all know . if i have a record having title "swimming pools georgia" then i think its similar to previous one approx 70% . I just want to dig out the algorithm who can help me to fetch these similar records with scoring .

Is any way to find similar records like this in ES ?

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [August 7, 2019, 11:46am UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/6 "2019-08-07T11:46:50Z")

</div>

Sounds like the [more like this](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html) feature. Does that work for you?

---

<div class="post-metadata">

**Author:** ![Dawood\_Siddiq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dawood_siddiq/32/51756_2.png) [@Dawood\_Siddiq](https://discuss.elastic.co/u/Dawood_Siddiq)\
**Post date:** [August 7, 2019, 11:50am UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/7 "2019-08-07T11:50:41Z")

</div>

No @xeraa . more like this feature doesn't work for me .

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [August 7, 2019, 12:44pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/8 "2019-08-07T12:44:17Z")

</div>

Ok, why not? We're not making much progress with that answer alone.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 4, 2019, 12:44pm UTC](https://discuss.elastic.co/t/dedup-records-according-to-percentage/194025/9 "2019-09-04T12:44:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.