# Remove redundancy of data

**URL:** <https://discuss.elastic.co/t/remove-redundancy-of-data/35273>\
**Category:** Elasticsearch\
**Created:** [November 23, 2015, 7:08am UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273 "2015-11-23T07:08:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amit2015](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@Amit2015](https://discuss.elastic.co/u/Amit2015)\
**Post date:** [November 23, 2015, 7:08am UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273/1 "2015-11-23T07:08:53Z")

</div>

Hi,

I have import data into elasticSearch and I havn't given unique ID. I want to remove redundant data. Please suggest how can I do that????

---

<div class="post-metadata">

**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\
**Post date:** [November 23, 2015, 2:15pm UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273/2 "2015-11-23T14:15:35Z")

</div>

The problem you describe sounds like a deduplication problem. Your favourite search engine will give you several posts on approaches how to do that.

One question though: I guess simply re-indexing and avoiding to import the duplicates is not an option for you?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [November 23, 2015, 2:32pm UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273/3 "2015-11-23T14:32:27Z")

</div>

> [@mainec](#):
>
> One question though: I guess simply re-indexing and avoiding to import the duplicates is not an option for you?

I think its worth mentioning that deletes have to be cleaned out of the index at some point so sometimes creating a new index and just indexing what you want is going to be faster. In this case it really works similarly to a relational database.

---

<div class="post-metadata">

**Author:** ![Amit2015](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@Amit2015](https://discuss.elastic.co/u/Amit2015)\
**Post date:** [November 23, 2015, 5:32pm UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273/4 "2015-11-23T17:32:01Z")

</div>

Hi,

I am using python to import csv file, which has no unique ID and File size also big.. So can't re-indexing.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 25, 2015, 6:57am UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273/5 "2015-11-25T06:57:15Z")

</div>

> I am using python to import csv file, which has no unique ID and File size also big.. So can't re-indexing.

I don't see why not. If you still have the CSV file you should be able to rerun the Python script to reindex, right? Changing the script to keep track of duplicates by hashing each line (or select fields from each line) and storing those hashes in a set should be pretty easy.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:36pm UTC](https://discuss.elastic.co/t/remove-redundancy-of-data/35273/6 "2017-07-05T23:36:04Z")

</div>