# Indexing-time document deduplication **URL:** \ **Category:** Elasticsearch\ **Created:** [June 8, 2011, 11:52pm UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579 "2011-06-08T23:52:22Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 8, 2011, 11:52pm UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/1 "2011-06-08T23:52:22Z")
Hello, What's the best way to deal deduplication of documents in the cluster? Imagine the same document comes down some pipe to the app that indexes docs into ES. Imagine there is no unique and persistent document ID one could use to quickly query the cluster to see if such a doc already exist. How would one detect that the same doc already exists in order to avoid indexing this document twice? Thanks, Otis ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 9, 2011, 1:07am UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/2 "2011-06-09T01:07:19Z")
I think at that point, you would only be able to query for the existence of a prior indexed document using the other fields in the document which would uniquely identify the document. It might be quicker to index the document again. 🙂 On Wed, Jun 8, 2011 at 7:52 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hello, > > What's the best way to deal deduplication of documents in the > cluster? Imagine the same document comes down some pipe to the app > that indexes docs into ES. Imagine there is no unique and persistent > document ID one could use to quickly query the cluster to see if such > a doc already exist. How would one detect that the same doc already > exists in order to avoid indexing this document twice? > > Thanks, > Otis ---
**Author:** ![fashionalwallet](https://avatars.discourse-cdn.com/v4/letter/f/839c29/32.png) [@fashionalwallet](https://discuss.elastic.co/u/fashionalwallet)\ **Post date:** [June 9, 2011, 8:53am UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/3 "2011-06-09T08:53:42Z")
- deleted - ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [June 9, 2011, 11:47am UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/4 "2011-06-09T11:47:37Z")
Otis, deduplication is a complicate task. But the main idea is to query with 'more like this' or 'fuzzy like this' for existing documents. You can take a look here: [http://zmievski.org/2011/03/duplicates-detection-with-elasticsearch](http://zmievski.org/2011/03/duplicates-detection-with-elasticsearch) Or at jetwick where I implemented this for tweets (via my own 'more like this' and afterwards a jaccard index check): > This has a large impact on indexing performance. At this time I implemented this with a refresh call after bulk indexing, but now I have changed this to avoid the call with the help of the versioning feature. But if you want faster duplicate queries (which is especially difficult for large documents) you'll need to use a local sensitive hashing technics: > **[Locality-sensitive hashing](https://en.wikipedia.org/wiki/Locality-sensitive_hashing)** > > Locality-sensitive hashing (LSH) reduces the dimensionality of high-dimensional data. LSH hashes input items so that similar items map to the same “buckets” with high probability (the number of buckets being much smaller than the universe of possible input items). LSH differs from conventional and cryptographic hash functions because it aims to maximize the probability of a “collision” for similar items. Locality-sensitive hashing has much in common with data clustering and nearest neighbor se Ha... which requires extra space (and memory). Regards, Peter. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [June 9, 2011, 7:15pm UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/5 "2011-06-09T19:15:01Z")
Or did you mean the dedup of Solr [http://wiki.apache.org/solr/Deduplication](http://wiki.apache.org/solr/Deduplication) ? It should work to grab the sources and use this for ES (as solr uses an the textsignature algo borrowed from tika, if I remember correctly). ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [June 9, 2011, 8:09pm UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/6 "2011-06-09T20:09:30Z")
If you want to do deduplication you need to be able to define equality between two documents. Ideally your documents should contain a unique key to compare on. If there are none you could compute such a unique key using, for example, some hashing over your content (md5/sha1 over some field(s)). Depending on the rate at which you insert new documents, you will most probably want to do you deduplication before the indexing phase in ES. For this you can keep the list of inserted unique IDs in memory, in a bloom filter for example. At this point it really depends on your architecture/use-case but there are many alternatives to avoid hitting ES for a duplicate check before indexing every document. If you want better suggestions, give us a little more context on your ES setup and user-case. Colin On Wed, Jun 8, 2011 at 7:52 PM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hello, > > What's the best way to deal deduplication of documents in the > cluster? Imagine the same document comes down some pipe to the app > that indexes docs into ES. Imagine there is no unique and persistent > document ID one could use to quickly query the cluster to see if such > a doc already exist. How would one detect that the same doc already > exists in order to avoid indexing this document twice? > > Thanks, > Otis ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:04am UTC](https://discuss.elastic.co/t/indexing-time-document-deduplication/4579/7 "2017-07-06T04:04:12Z")