# \[Ingest Pipeline\] Drop fields if duplicate

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-drop-fields-if-duplicate/234971>\
**Category:** Elasticsearch\
**Created:** [May 29, 2020, 6:41pm UTC](https://discuss.elastic.co/t/ingest-pipeline-drop-fields-if-duplicate/234971 "2020-05-29T18:41:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fernando\_Ruben\_Otero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fernando_ruben_otero/32/64932_2.png) [@Fernando\_Ruben\_Otero](https://discuss.elastic.co/u/Fernando_Ruben_Otero)\
**Post date:** [May 29, 2020, 6:41pm UTC](https://discuss.elastic.co/t/ingest-pipeline-drop-fields-if-duplicate/234971/1 "2020-05-29T18:41:49Z")

</div>

Is it possible to drop a document if there's another document with the same \_source, or with the same subset of fields from \_source?

```auto
    "processors" : [
    {
      "drop": {
        "if": "ctx._source == documents[ctx._id]._source"
       }
    }
  ]

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 31, 2020, 3:29pm UTC](https://discuss.elastic.co/t/ingest-pipeline-drop-fields-if-duplicate/234971/2 "2020-05-31T15:29:06Z")

</div>

The processors generally work within the context of a simgle document, so do not have access to other documents already in the index. If you are looking to avoid duplicates you can do this by assigning a predictable ID that will cause an update when the duplicate arrives. This is described [in this old blog post](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2020, 3:29pm UTC](https://discuss.elastic.co/t/ingest-pipeline-drop-fields-if-duplicate/234971/3 "2020-06-28T15:29:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.