# Query to get one doc out of its duplicate

**URL:** <https://discuss.elastic.co/t/query-to-get-one-doc-out-of-its-duplicate/246396>\
**Category:** Kibana\
**Created:** [August 26, 2020, 6:48am UTC](https://discuss.elastic.co/t/query-to-get-one-doc-out-of-its-duplicate/246396 "2020-08-26T06:48:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manikantaks](https://avatars.discourse-cdn.com/v4/letter/m/57b2e6/32.png) [@Manikantaks](https://discuss.elastic.co/u/Manikantaks)\
**Post date:** [August 26, 2020, 6:48am UTC](https://discuss.elastic.co/t/query-to-get-one-doc-out-of-its-duplicate/246396/1 "2020-08-26T06:48:52Z")

</div>

I have ELK set up . In specific index there are around 70Lac records , out of which some are duplicates . I need to create a filter in kibana so that only uniq records should be return. I need to check for two field for the records uniqness . Example , phone number and city fields of Document 1 and Document 2 are same then we consider those Documents are same , in this case query should return any one of these to Documents. here Id is auto generated we can't use that for uniqness. Need to help on this.

---

<div class="post-metadata">

**Author:** ![Nathan\_Reese](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_reese/32/84829_2.png) [@Nathan\_Reese](https://discuss.elastic.co/u/Nathan_Reese)\
**Post date:** [August 28, 2020, 2:02pm UTC](https://discuss.elastic.co/t/query-to-get-one-doc-out-of-its-duplicate/246396/2 "2020-08-28T14:02:25Z")

</div>

Why does your data have duplicates? Are the duplicates needed for other use cases? How is your data getting ingested? I would recommend removing duplicates during ingest. You can specify your own `_id` field. Maybe it makes sense to make this field be the concatenation of phone number and city.

---

<div class="post-metadata">

**Author:** ![Manikantaks](https://avatars.discourse-cdn.com/v4/letter/m/57b2e6/32.png) [@Manikantaks](https://discuss.elastic.co/u/Manikantaks)\
**Post date:** [September 1, 2020, 7:33am UTC](https://discuss.elastic.co/t/query-to-get-one-doc-out-of-its-duplicate/246396/3 "2020-09-01T07:33:39Z")

</div>

@Nathan_Reese`Data already ingested. Now I need to read data based on the above condition . I need to do operations already existing data.`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 29, 2020, 7:33am UTC](https://discuss.elastic.co/t/query-to-get-one-doc-out-of-its-duplicate/246396/4 "2020-09-29T07:33:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.