# Deduplicating data in ElasticSearch

**URL:** <https://discuss.elastic.co/t/deduplicating-data-in-elasticsearch/96274>\
**Category:** Elasticsearch\
**Created:** [August 8, 2017, 1:24pm UTC](https://discuss.elastic.co/t/deduplicating-data-in-elasticsearch/96274 "2017-08-08T13:24:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rafay\_Khan](https://avatars.discourse-cdn.com/v4/letter/r/4491bb/32.png) [@Rafay\_Khan](https://discuss.elastic.co/u/Rafay_Khan)\
**Post date:** [August 8, 2017, 1:24pm UTC](https://discuss.elastic.co/t/deduplicating-data-in-elasticsearch/96274/1 "2017-08-08T13:24:48Z")

</div>

I am facing a challenge where I need to reduce my indexed data, by clustering similar items into one. Actually, somewhat deduplicating data. For eg.  
\> -----------------------------------------------------------------------------------------------------------------------------------------------

> ```
> Product Name | Category | Product Group | Attributes
> -----------------------------------------------------------------------------------------------------------------------------------------------
> 1- Board Marker - Blue | Stationary | White Board Markers | Color = Blue, Type = Board
> 2- Blue Board Marker | Stationary | White Board Markers | Color = Blue, Type = Board 
> ---------------------------------------------------------------------------------------------------------------------------------------------------
> 
> ```

In the new indexed data I need to store the data as:  
\> -----------------------------------------------------------------------------------------------------------------------------------------------

> ```
> Product Name | Category | Product Group |  
> -----------------------------------------------------------------------------------------------------------------------------------------------
> 1- Board Marker - Blue | Stationary | White Board Markers | ...
> 
> ```

I did see @nfantone's similar question on "Clustering data on elastic Search index"([Clustering data on Elasticsearch index](https://discuss.elastic.co/t/clustering-data-on-elasticsearch-index/53262)). Some of the replies there mentioned about the :

> There is a proposal for a fingerprinting inget processor here: [https://github.com/elastic/elasticsearch/issues/1693811](https://github.com/elastic/elasticsearch/issues/1693811) that you might find interesting as well.

and secondly, @Mark_Harwood explaining about:

> Automate some batch data de-duplication process with minimal human intervention.

> Requires more rigour and emphasises precision over recall. Each merge operation has to be something that can be trusted for the algorithm to iterate on without human intervention. A person cannot be allowed to over-link and become his brother then their father etc through a steady accumulation of weakly linked properties.  
> Large-scale iterative entity resolution can be achieved but uses non-fuzzy keys and lots of different ways of composing the keys. I demoed this 37 minutes into this presentation at elasticon: [https://www.elastic.co/elasticon/conf/2016/sf/graph-capabilities-in-the-elastic-stack3](https://www.elastic.co/elasticon/conf/2016/sf/graph-capabilities-in-the-elastic-stack3)

Can anyone explain how to go about doing these, as the fingerprint feature seems mature enough, since the question was posted, and implementing Mark\_Harwood's approach(I have seen the linked presentation).

TL,DR: Find items which are at least 90% similar, on properties such as Product Name, Category , Product Group, (some)Attributes, and generate a new index to store them. I am new to Elasticsearch and a noob at this.

---

<div class="post-metadata">

**Author:** ![Rafay\_Khan](https://avatars.discourse-cdn.com/v4/letter/r/4491bb/32.png) [@Rafay\_Khan](https://discuss.elastic.co/u/Rafay_Khan)\
**Post date:** [August 15, 2017, 1:52pm UTC](https://discuss.elastic.co/t/deduplicating-data-in-elasticsearch/96274/2 "2017-08-15T13:52:48Z")

</div>

Any suggestions?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 12, 2017, 1:53pm UTC](https://discuss.elastic.co/t/deduplicating-data-in-elasticsearch/96274/3 "2017-09-12T13:53:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.