[Ingest Pipeline] Drop fields if duplicate

[Fernando_Ruben_Otero]

Is it possible to drop a document if there's another document with the same _source, or with the same subset of fields from _source?

    "processors" : [
    {
      "drop": {
        "if": "ctx._source == documents[ctx._id]._source"
       }
    }
  ]

[Christian_Dahlqvist]

The processors generally work within the context of a simgle document, so do not have access to other documents already in the index. If you are looking to avoid duplicates you can do this by assigning a predictable ID that will cause an update when the duplicate arrives. This is described in this old blog post.