# Remove duplicate / multiple data records in Elastic

**URL:** <https://discuss.elastic.co/t/remove-duplicate-multiple-data-records-in-elastic/282569>\
**Category:** Elasticsearch\
**Created:** [August 26, 2021, 11:59am UTC](https://discuss.elastic.co/t/remove-duplicate-multiple-data-records-in-elastic/282569 "2021-08-26T11:59:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cf4455](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@cf4455](https://discuss.elastic.co/u/cf4455)\
**Post date:** [August 26, 2021, 11:59am UTC](https://discuss.elastic.co/t/remove-duplicate-multiple-data-records-in-elastic/282569/1 "2021-08-26T11:59:44Z")

</div>

Hi, due to an error in the program that writes data records to the Elastic, there are now a lot of entries in our Elastic, all with exactly the same value in the "time" attribute. For example, there are 50 data records in which the value "Aug 5, 2021 @ 10:39:37.720" is in the "time" field (The "time" field is filled by a program of us, not by the elastic). In this way there are many more entries where the time is exactly the same to the millisecond. Now I would remove all of the multiple entries so that there is only 1 entry for a time. That theoretically 2 different entries are displayed at the same time can be ignored. At the moment there are around 630,000 records in the Elastic, around 300,000 are probably double or multiple.

Unfortunately, I am not very familiar with Elastic Query. How can all duplicate or multiple entries be identified and removed?

Is there also a way to hide the duplicate entries, e.g. using a filter in the Kibana?

Elastic Version is 7.11.2.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 26, 2021, 3:33pm UTC](https://discuss.elastic.co/t/remove-duplicate-multiple-data-records-in-elastic/282569/2 "2021-08-26T15:33:08Z")

</div>

See [Delete by query API | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/docs-delete-by-query.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 23, 2021, 3:33pm UTC](https://discuss.elastic.co/t/remove-duplicate-multiple-data-records-in-elastic/282569/3 "2021-09-23T15:33:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.