Pipeline not applied to all documents

Elastic Stack Elasticsearch
1

Hi,

I implemented a script pipeline to finally convert some fields programmatically. First I wanted to test it and it turned out that the pipeline is not applied to all documents but only to a few. This is my pipeline:

PUT _ingest/pipeline/survey_name
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "script" : {
        "lang": "painless",
        "on_failure": [
          {
            "set" : {
              "field" : "error",
              "value" : "field \"foo\" does not exist, cannot rename to \"bar\""
            }
          }
        ],
        "source": """
        ctx.search = '111111111111';
        """
      }
    }
  ]
}

The index contains these fields:

{
  "_routing": {
    "required": true
  },
  "properties": {
    "entity": { "type": "keyword" },
    "search": { "type": "keyword" },
    "error": {"type":  "keyword"},
  ...
}

In the documents where the search field is not going to be set the error field is not set either. I don't see any patterns on the documents whether it works or not. Since the inserting is done by Queue-consumers the doucments are inserted closely to in parallel - Could this maybe a problem?

Can anyone help here?

EDIT: Please don't care about the namings - This is mostly copy & pasted.

2

EDIT: I investigated further and it seems that the pipeline is only applied on every second document.. WTH?

3

Hi @Marc_Zahn,

can you provide sample indexing requests that exhibit the "every second document" behavior"?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.