Append processor overwrites, doesn't append

Elastic Stack Elasticsearch
1

I'm playing with the append processor, but it seems to be overwriting the array each time, not appending.

PUT /_ingest/pipeline/append
{
  "description": "Append",
  "processors": [
    {
      "append": {
        "field": "array",
        "value": [
          "{{value}}"
        ]
      }
    }
  ]
}

Then I test it out:

PUT localhost:9200/test/_doc/1?pipeline=append
{"value": 1}

_source:

{
    "array": ["1"],
    "value": 1
}

Then,

PUT localhost:9200/test/_doc/1?pipeline=append
{"value": 2}

_source:

{
    "array": ["2"],
    "value": 2
}

Shouldn't the result have "array": ["1", "2"]? If not, how is this append processor designed to be used?

2

The way the append processor works is that it appends the values in value to the field specified in field, and it operates independently on each document. So if I use the _simulate API:

POST _ingest/pipeline/append/_simulate
{
  "docs": [
    {
      "_source": {
        "value": 1,
        "array": [3,2]
      }
    },
    {
      "_source": {
        "value": 6,
        "array": [4,5]
      }
    }
  ]
}

I get back this result:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "array" : [3, 2, "1"],
          "value" : 1
        },
        "_ingest" : {
          "timestamp" : "2019-02-25T21:51:30.282Z"
        }
      }
    },
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "array" : [4, 5, "6"],
          "value" : 6
        },
        "_ingest" : {
          "timestamp" : "2019-02-25T21:51:30.282Z"
        }
      }
    }
  ]
}

Does that help?

Edit to expand a bit: Since you're not providing an array field in the input documents, the processor is creating a new, empty array and appending value to that empty array.

3

Argh. That makes sense. Upon reading the description in the docs, I had hoped that each time you index the doc it appended the value to an array rather than overwriting it.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.