# How to reindex distinct data in elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943>\
**Category:** Elasticsearch\
**Created:** [September 7, 2016, 6:16am UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943 "2016-09-07T06:16:09Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![gayathri](https://avatars.discourse-cdn.com/v4/letter/g/838e76/32.png) [@gayathri](https://discuss.elastic.co/u/gayathri)\
**Post date:** [September 7, 2016, 6:16am UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943/1 "2016-09-07T06:16:09Z")

</div>

Hi Team,

I am able to reindex the data in elasticsearch using:

- curl -XPOST [http://localhost:9200/\_reindex?pretty](http://localhost:9200/_reindex?pretty) -d'  
{

I have many duplicate log entries in my index..I want to reindex by removing duplicate entries.  
Could u please suggest any method?

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [September 8, 2016, 10:40am UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943/2 "2016-09-08T10:40:48Z")

</div>

The scripts that I use for entity-centric indexing [1] sort content in a source index by a common key and consolidate multiple docs into a update on a single document in the target index. The "pull" from the source index and the "push" to the target index are both done using the respective bulk APIs.  
Your use case is slightly different in that you want to _insert_ a single doc in the target index rather than update one but you should be able to adapt the included python script with few changes.

[1] [http://bit.ly/entcent](http://bit.ly/entcent)

---

<div class="post-metadata">

**Author:** ![kstaken](https://avatars.discourse-cdn.com/v4/letter/k/e68b1a/32.png) [@kstaken](https://discuss.elastic.co/u/kstaken)\
**Post date:** [September 8, 2016, 9:40pm UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943/3 "2016-09-08T21:40:29Z")

</div>

Just a suggestion, come up with a way to create a key from the data in the log entries and then use create requests to save the data into a new index under that key. If there are duplicate entries then only the first will be indexed and the duplicates will be dropped. Unfortunately, I don't know if the reindex api supports this.

Kimbro

---

<div class="post-metadata">

**Author:** ![gayathri](https://avatars.discourse-cdn.com/v4/letter/g/838e76/32.png) [@gayathri](https://discuss.elastic.co/u/gayathri)\
**Post date:** [September 13, 2016, 6:44am UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943/4 "2016-09-13T06:44:57Z")

</div>

Is it possible to remove the duplicate log entries from an existing index ?

---

<div class="post-metadata">

**Author:** ![kstaken](https://avatars.discourse-cdn.com/v4/letter/k/e68b1a/32.png) [@kstaken](https://discuss.elastic.co/u/kstaken)\
**Post date:** [September 14, 2016, 3:52pm UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943/5 "2016-09-14T15:52:54Z")

</div>

There's no easy way to do that. You'd still have to walk the entire index and figure out what are duplicates then delete them. It will be a lot simpler to simply reindex and drop the duplicates in that process.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:20pm UTC](https://discuss.elastic.co/t/how-to-reindex-distinct-data-in-elasticsearch/59943/6 "2017-07-05T22:20:13Z")

</div>