# Prevent duplicate content in search

**URL:** <https://discuss.elastic.co/t/prevent-duplicate-content-in-search/138535>\
**Category:** Elasticsearch\
**Created:** [July 4, 2018, 10:30am UTC](https://discuss.elastic.co/t/prevent-duplicate-content-in-search/138535 "2018-07-04T10:30:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yurii](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yurii/32/32948_2.png) [@Yurii](https://discuss.elastic.co/u/Yurii)\
**Post date:** [July 4, 2018, 10:30am UTC](https://discuss.elastic.co/t/prevent-duplicate-content-in-search/138535/1 "2018-07-04T10:30:39Z")

</div>

Hey! I faced a problem when I tried to get a related post. There are few posts which are identical to each other or very similar.  
So when I try to get the related posts there some cases when in a related post are few duplicated posts which have identical content relatively each other.  
How I can solve this problem?

---

<div class="post-metadata">

**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\
**Post date:** [July 7, 2018, 9:06am UTC](https://discuss.elastic.co/t/prevent-duplicate-content-in-search/138535/2 "2018-07-07T09:06:33Z")

</div>

> [@Yurii](#):
>
> There are few posts which are identical to each other

There are a few options here:

- try to detect posts are identical at index time and discard the exact-duplicates. For this ppl. usually do some sort identity fingerprinting (like md5 hashing or similar) on the content and either compare those hashes to already seen hashed in a fast in-memory datastructure. If that is not feasible, you could store that info in Elasticsearch itself but then would have to do an additional query to check if the hash/document is already present
- you could do the same as above but with a batch job, indexing the fingerprint value, periodically scan for duplicates and discard the extra ones

> [@Yurii](#):
>
> There are few posts which are [...] very similar.

Thats a harder problem. There is a whole research area dedicated to find near-duplicates. What counts as "very similar" is also very domain specific. You might get away with simple fuzzy searches, but usually spotting this involves more complicated methods, mostly shingling algorithms. In a project I once did we implemented an algorithm by [Broder](http://cs.brown.edu/courses/cs253/papers/nearduplicate.pdf), but there might be other aproaches. No silver bullets here as far as I know...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2018, 9:06am UTC](https://discuss.elastic.co/t/prevent-duplicate-content-in-search/138535/3 "2018-08-04T09:06:47Z")

</div>

Closed on August 4, 2018, 9:06am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.