Pipeline not applied to all documents


I implemented a script pipeline to finally convert some fields programmatically. First I wanted to test it and it turned out that the pipeline is not applied to all documents but only to a few. This is my pipeline:

PUT _ingest/pipeline/survey_name
  "description" : "describe pipeline",
  "processors" : [
      "script" : {
        "lang": "painless",
        "on_failure": [
            "set" : {
              "field" : "error",
              "value" : "field \"foo\" does not exist, cannot rename to \"bar\""
        "source": """
        ctx.search = '111111111111';

The index contains these fields:

  "_routing": {
    "required": true
  "properties": {
    "entity": { "type": "keyword" },
    "search": { "type": "keyword" },
    "error": {"type":  "keyword"},

In the documents where the search field is not going to be set the error field is not set either. I don't see any patterns on the documents whether it works or not. Since the inserting is done by Queue-consumers the doucments are inserted closely to in parallel - Could this maybe a problem?

Can anyone help here?

EDIT: Please don't care about the namings - This is mostly copy & pasted.

EDIT: I investigated further and it seems that the pipeline is only applied on every second document.. WTH?

Hi @Marc_Zahn,

can you provide sample indexing requests that exhibit the "every second document" behavior"?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.