# Removal of duplicate based on some condition

**URL:** <https://discuss.elastic.co/t/removal-of-duplicate-based-on-some-condition/134862>\
**Category:** Elasticsearch\
**Created:** [June 6, 2018, 7:04pm UTC](https://discuss.elastic.co/t/removal-of-duplicate-based-on-some-condition/134862 "2018-06-06T19:04:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ssenthil416](https://avatars.discourse-cdn.com/v4/letter/s/8e8cbc/32.png) [@ssenthil416](https://discuss.elastic.co/u/ssenthil416)\
**Post date:** [June 6, 2018, 7:04pm UTC](https://discuss.elastic.co/t/removal-of-duplicate-based-on-some-condition/134862/1 "2018-06-06T19:04:07Z")

</div>

We have millions of rows as part of elastic, in that we found many duplicate IDs, we want to clean it up  
We use Go Elastic library `https://github.com/olivere/elastic`

We have query to first 10000 record to return duplicate  
agg := elastic.NewTermsAggregation().Field("\_id").MinDocCount(2).Size(10000)  
search = client.Search("employee")  
search = search.Aggregation("myterm", agg)  
sr, err := search.Do(context.TODO())

but we want this query to run for all millions of record we have and we want to get use of scroll API of elastic for it....

Can any one please help us

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 4, 2018, 7:04pm UTC](https://discuss.elastic.co/t/removal-of-duplicate-based-on-some-condition/134862/2 "2018-07-04T19:04:10Z")

</div>

Closed on July 4, 2018, 7:04pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.