# Finding duplicate documents or its count based on some field names **URL:** \ **Category:** Elasticsearch\ **Created:** [December 28, 2013, 5:16am UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041 "2013-12-28T05:16:16Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![narinder\_izap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narinder_izap/32/803_2.png) [@narinder\_izap](https://discuss.elastic.co/u/narinder_izap)\ **Post date:** [December 28, 2013, 5:16am UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041/1 "2013-12-28T05:16:16Z")
Hi All, ``` I need to know, if Elasticsearch has some feature to find the ``` duplicate documents or documents counts if I want to see how many documents are having same values against two or more fields. I can do that for one field using facets, but what if I need to do it against more than one field. For Example : Suppose I have following doc in Es doc 1 : { name : abc age:22 country:usa gender:male } doc 2 : { name:xyz age:27 country:usa gender:male } doc 3: { name:xyz age:22 country:india gender:female } doc 4 { name:abc age:22 country:usa gender:female } So now my requirement is to find all doc having same age and same country, So that doc1 and doc4 are duplicate for me, OR In simple words, I want to have unique clause on a single fields or composite fields key. Is this possible?? Please let me know if its possible using Elasticsearch, as I think it is very important feature for me. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1988ddb0-9bae-4263-b262-3c84f4445fa8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1988ddb0-9bae-4263-b262-3c84f4445fa8%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 28, 2013, 1:14pm UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041/2 "2013-12-28T13:14:50Z")
If you can use 1.0.0.Beta2, aggregations might be a solution. Demo: > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGExLpeiBcMxcemFDo9oqOLjCj9zq4Db8\_mNrp8n%2BV3-w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGExLpeiBcMxcemFDo9oqOLjCj9zq4Db8_mNrp8n%2BV3-w%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [December 30, 2013, 5:07pm UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041/3 "2013-12-30T17:07:33Z")
More Like This could work, especially if using non-analyzed fields: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -- Ivan On Sat, Dec 28, 2013 at 5:14 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > If you can use 1.0.0.Beta2, aggregations might be a solution. > > Demo: > > [Elasticsearch aggregations demo · GitHub](https://gist.github.com/jprante/8159379) > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGExLpeiBcMxcemFDo9oqOLjCj9zq4Db8\_mNrp8n%2BV3-w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGExLpeiBcMxcemFDo9oqOLjCj9zq4Db8_mNrp8n%2BV3-w%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAmk5sUPgDd\_1eA\_pF%2BU%2B0ud0C9n4xVgsmHZmjvFM\_\_gg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAmk5sUPgDd_1eA_pF%2BU%2B0ud0C9n4xVgsmHZmjvFM__gg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [December 31, 2013, 12:57pm UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041/4 "2013-12-31T12:57:27Z")
Hi, You can check this : > **[GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity...](https://github.com/yannbrrd/elasticsearch-entity-resolution)** > > Elasticsearch entity resolution plugin based on Duke - GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity resolution plugin based on Duke Le samedi 28 décembre 2013 06:16:16 UTC+1, Narinder Kaur a écrit : > Hi All, > > ``` > I need to know, if Elasticsearch has some feature to find the > > ``` > > duplicate documents or documents counts if I want to see how many documents > are having same values against two or more fields. I can do that for one > field using facets, but what if I need to do it against more than one > field. For Example : Suppose I have following doc in Es > > doc 1 : > > { > name : abc > age:22 > country:usa > gender:male > } > > doc 2 : > > { > name:xyz > age:27 > country:usa > gender:male > } > > doc 3: > > { > name:xyz > age:22 > country:india > gender:female > } > > doc 4 > { > name:abc > age:22 > country:usa > gender:female > } > > So now my requirement is to find all doc having same age and same country, > So that doc1 and doc4 are duplicate for me, OR In simple words, I want > to have unique clause on a single fields or composite fields key. Is this > possible?? > > Please let me know if its possible using Elasticsearch, as I think it is > very important feature for me. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7d4ebe89-f777-499c-a215-f794c33d88a3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d4ebe89-f777-499c-a215-f794c33d88a3%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 1, 2014, 8:20pm UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041/5 "2014-01-01T20:20:39Z")
Hey, another very simple solution could be a terms facet, using a script field, which simply concatenates the two fields you want to check for. See > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. --Alex On Tue, Dec 31, 2013 at 1:57 PM, Yann Barraud [yann.barraud@gmail.com](mailto:yann.barraud@gmail.com)wrote: > Hi, > > You can check this : > > [GitHub - YannBrrd/elasticsearch-entity-resolution: Elasticsearch entity resolution plugin based on Duke](http://github.com/yannbrrd/elasticsearch-entity-resolution) > > Le samedi 28 décembre 2013 06:16:16 UTC+1, Narinder Kaur a écrit : > > > Hi All, > > > > ``` > > I need to know, if Elasticsearch has some feature to find the > > > > ``` > > > > duplicate documents or documents counts if I want to see how many documents > > are having same values against two or more fields. I can do that for one > > field using facets, but what if I need to do it against more than one > > field. For Example : Suppose I have following doc in Es > > > > doc 1 : > > > > { > > name : abc > > age:22 > > country:usa > > gender:male > > } > > > > doc 2 : > > > > { > > name:xyz > > age:27 > > country:usa > > gender:male > > } > > > > doc 3: > > > > { > > name:xyz > > age:22 > > country:india > > gender:female > > } > > > > doc 4 > > { > > name:abc > > age:22 > > country:usa > > gender:female > > } > > > > So now my requirement is to find all doc having same age and same > > country, So that doc1 and doc4 are duplicate for me, OR In simple words, > > I want to have unique clause on a single fields or composite fields key. Is > > this possible?? > > > > Please let me know if its possible using Elasticsearch, as I think it is > > very important feature for me. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7d4ebe89-f777-499c-a215-f794c33d88a3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d4ebe89-f777-499c-a215-f794c33d88a3%40googlegroups.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9u46nmj7Kzx0WZ0zUJ7xeT4e00HAh8Ce7j5DrnVY4uEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9u46nmj7Kzx0WZ0zUJ7xeT4e00HAh8Ce7j5DrnVY4uEg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:58am UTC](https://discuss.elastic.co/t/finding-duplicate-documents-or-its-count-based-on-some-field-names/15041/6 "2017-07-06T01:58:44Z")