[Ingest Pipeline] Drop fields if duplicate

Is it possible to drop a document if there's another document with the same _source, or with the same subset of fields from _source?

    "processors" : [
      "drop": {
        "if": "ctx._source == documents[ctx._id]._source"

The processors generally work within the context of a simgle document, so do not have access to other documents already in the index. If you are looking to avoid duplicates you can do this by assigning a predictable ID that will cause an update when the duplicate arrives. This is described in this old blog post.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.