Ingest pipeline set processor override property

Elastic Stack Elasticsearch
1

Hi,

I have difficulties in using the "override" option of the SET processor. The use case is the following:

  1. I'm creating a document like this:

    POST test_index/_doc
{
  "foo": "foo",
  "bar": "bar"
}

  2. Sometimes I have cases when an update for an existing document comes with a value of null for field "foo" or "bar", like:

    PUT test_index/_doc/doc_id
{
   "foo": null,
   "bar": "bar"
 }

What I want is: if field foo or bar is null and for the existing document there's already a non-null value for the same field, do not override the value by setting it to null. According to https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html, the override field has the default value true and its description is:

If processor will update fields with pre-existing non-null-valued field. When set to false , such fields will not be touched.

This is exactly what I want. I created the pipeline like this:

PUT _ingest/pipeline/preventNullValuesOverriding
{
  "description" : "ignore null values for foo and bar",
  "processors" : [
    {
      "set" : {
        "field": "foo",
        "value": "default",
        "override": false
      }
    },
    {
      "set" : {
        "field": "bar",
        "value": "default",
        "override": false
      }
    }
  ]
}

I've put the value of "default" because I'm forced to do put a value. I then attached the processor to the test_index index.

But then if I modify an existing document by setting foo or bar to null, the merge result will be "default" for that field.

Can somebody point me to how I can achieve my desired behaviour?

thanks a lot!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.