# Similar document detection **URL:** \ **Category:** Elasticsearch\ **Created:** [February 18, 2017, 3:04pm UTC](https://discuss.elastic.co/t/similar-document-detection/75604 "2017-02-18T15:04:05Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![avaneeshr](https://avatars.discourse-cdn.com/v4/letter/a/85f322/32.png) [@avaneeshr](https://discuss.elastic.co/u/avaneeshr)\ **Post date:** [February 18, 2017, 3:04pm UTC](https://discuss.elastic.co/t/similar-document-detection/75604/1 "2017-02-18T15:04:05Z")
Hi! I have a large 100GB index which contains webpages (url, content, etc.). A lot of these pages are from the same domain and are very similar (like 90% similar, with just a few modifications). I am trying to implement a full-text search over these documents and currently have done de-duplication using Jaccard coefficient as a postprocessing step in my Java app. This eliminates the duplicates, but i want to do this using native elasticsearch methods, if possible (such as "dedup"). Is it possible to get rid of the deduplication logic from my app and get rid of similar results from elasticsearch itself? using ver 2.4.0 Thanks in advance ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 18, 2017, 10:10pm UTC](https://discuss.elastic.co/t/similar-document-detection/75604/2 "2017-02-18T22:10:29Z")
There's not really anything like this natively. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 18, 2017, 10:10pm UTC](https://discuss.elastic.co/t/similar-document-detection/75604/3 "2017-03-18T22:10:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.