# Any idea to remove the duplicates from the search results? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 26, 2014, 7:16am UTC](https://discuss.elastic.co/t/any-idea-to-remove-the-duplicates-from-the-search-results/19975 "2014-09-26T07:16:04Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Peiyong\_Lin](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@Peiyong\_Lin](https://discuss.elastic.co/u/Peiyong_Lin)\ **Post date:** [September 26, 2014, 7:16am UTC](https://discuss.elastic.co/t/any-idea-to-remove-the-duplicates-from-the-search-results/19975/1 "2014-09-26T07:16:04Z")
Hi all, I have some documents that look like: { "name": "", "image\_url: "", "simHash": "" } And there are many duplicate documents with the same `simHash` value, any idea to remove the duplicates instead of removing them in indexing time? I have seen solutions here: [http://stackoverflow.com/questions/24080846/removing-duplicates-from-search-results](http://stackoverflow.com/questions/24080846/removing-duplicates-from-search-results) However, I would like to know if it is possible that I can write code in Elasticsearch to do this job. many thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fb60e5b-8a88-4af3-b0c1-0c064b280c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fb60e5b-8a88-4af3-b0c1-0c064b280c7e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tugberk\_Ugurlu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tugberk_ugurlu/32/1250_2.png) [@Tugberk\_Ugurlu](https://discuss.elastic.co/u/Tugberk_Ugurlu)\ **Post date:** [September 26, 2014, 9:04am UTC](https://discuss.elastic.co/t/any-idea-to-remove-the-duplicates-from-the-search-results/19975/2 "2014-09-26T09:04:44Z")
Hi, You are probably looking for the same thing that I was looking for a few days ago: [https://groups.google.com/forum/#!searchin/elasticsearch/tugberk/elasticsearch/1uCQ7R8vCS8/-iRJLrdGGrYJ](https://groups.google.com/forum/#!searchin/elasticsearch/tugberk/elasticsearch/1uCQ7R8vCS8/-iRJLrdGGrYJ) Top hits aggregation [http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/search-aggregations-metrics-top-hits-aggregation.html#search-aggregations-metrics-top-hits-aggregation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/search-aggregations-metrics-top-hits-aggregation.html#search-aggregations-metrics-top-hits-aggregation) may be what you are looking for. I had a working query here: [https://groups.google.com/forum/#!msg/elasticsearch/1uCQ7R8vCS8/MKpbJjVM0SMJ](https://groups.google.com/forum/#!msg/elasticsearch/1uCQ7R8vCS8/MKpbJjVM0SMJ) but note that I wasn’t able to paginate over aggregation result. It seems not possible at first glance: [https://github.com/elasticsearch/elasticsearch/issues/4915](https://github.com/elasticsearch/elasticsearch/issues/4915) Tugberk From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Peiyong Lin Sent: Friday, September 26, 2014 10:16 AM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Any idea to remove the duplicates from the search results? Hi all, I have some documents that look like: { ``` "name": "", "image_url: "", "simHash": "" ``` } And there are many duplicate documents with the same `simHash` value, any idea to remove the duplicates instead of removing them in indexing time? I have seen solutions here: [http://stackoverflow.com/questions/24080846/removing-duplicates-from-search-results](http://stackoverflow.com/questions/24080846/removing-duplicates-from-search-results) However, I would like to know if it is possible that I can write code in Elasticsearch to do this job. many thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) . To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fb60e5b-8a88-4af3-b0c1-0c064b280c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fb60e5b-8a88-4af3-b0c1-0c064b280c7e%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/5fb60e5b-8a88-4af3-b0c1-0c064b280c7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fb60e5b-8a88-4af3-b0c1-0c064b280c7e%40googlegroups.com?utm_medium=email&utm_source=footer) . For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/055901cfd968%24eb502cc0%24c1f08640%24%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/055901cfd968%24eb502cc0%24c1f08640%24%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:59am UTC](https://discuss.elastic.co/t/any-idea-to-remove-the-duplicates-from-the-search-results/19975/3 "2017-07-06T00:59:45Z")