# Remove duplicate documents that have the same field value

**URL:** <https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604>\
**Category:** Logstash\
**Created:** [August 2, 2019, 5:54pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604 "2019-08-02T17:54:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![gabrielconte14](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabrielconte14/32/46713_2.png) [@gabrielconte14](https://discuss.elastic.co/u/gabrielconte14)\
**Post date:** [August 2, 2019, 5:54pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604/1 "2019-08-02T17:54:55Z")

</div>

I'm trying to retrieve data from MongoDB and remove documents that have the same values on a specific field.

For example: If the value of the field "device\_uuid" of multiple documents are equal, I would like to filter these documents with duplicated fields and remain with a unique document with that field value.

I've been trying to use fingerprint to do that job, but different documents are still being created in elasticsearch, even if the field "device\_uuid" are equal. The only case that it works is when all the fields are equal, then the filter is applied.

This is my code:

```
  filter {
    mutate {
      remove_field => ["_id"]
    }
    fingerprint {
      source => [device_uuid]
      target => "fingerprint"
      key => "78787878"
      method => "SHA1"
      concatenate_sources => true
    }
  }
 
  output {
      elasticsearch {
         index => "logstash_test"
         hosts => ["elastic_url_here"]
         document_id => "%{fingerprint}"
      }
 
 
   stdout{
     codec => rubydebug
   }
 }
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 2, 2019, 6:04pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604/2 "2019-08-02T18:04:11Z")

</div>

If you just look at the device\_uuid and fingerprint fields does the same device\_uuid result in more than one fingerprint?

---

<div class="post-metadata">

**Author:** ![gabrielconte14](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabrielconte14/32/46713_2.png) [@gabrielconte14](https://discuss.elastic.co/u/gabrielconte14)\
**Post date:** [August 2, 2019, 6:25pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604/3 "2019-08-02T18:25:02Z")

</div>

You are absolutely right! The document with that uuid was being overridden but the fingerprint was the same! It was actually working!  
😁

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [August 2, 2019, 6:47pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604/4 "2019-08-02T18:47:20Z")

</div>

Maybe I'm missing something here, but when an existing doc id is found, elasticsearch performs an update of the existing event and overwrites it with the new event that is coming in.

So if the original ingested document has `field1`=`Hello` and a `uuid` of `1`, and then later you ingest a document where `field1` = `World` which also has a `uuid` of `1`, then you lose the original event that contained `Hello`.

Is that what you are wanting to accomplish, to always update a given event with the latest version and lose previously ingested data?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2019, 6:47pm UTC](https://discuss.elastic.co/t/remove-duplicate-documents-that-have-the-same-field-value/193604/5 "2019-08-30T18:47:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.