How to rename a field within an array in ingest pipeline?

Elastic Stack Elasticsearch
1

I have created an ingest pipeline that is being used from my logstash output plugin for elasticsearch. This pipeline only does some renaming of fields.

Now, this 'rename' processor is working fine for leaf or nested fields but not for arrays fields. Can someone please help and correct me?

In my example, 'models' is the array for which rename processor is failing. What would be the syntax?

PUT _ingest/pipeline/rename_model
{
  "processors": [
    {
      "rename": {
        "field": "brand_id",
        "target_field": "brandId",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "version.date_year",
        "target_field": "version.year",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "models.model_id",
        "target_field": "models.modelId",
        "ignore_missing": true
      }
    }
  ]
}

POST _ingest/pipeline/rename_model/_simulate
{
  "docs": [
    {
      "_source": {
        "brand_id": "hello",
        "brand_name": "HI",
        "version": {
          "date_year": 2018
        },
        "models": [
          {
            "model_number": "Test1",
            "model_id": 123
          },
          {
            "model_number": "Test2",
            "model_id": 456
          }
        ]
      }
    }
  ]
}
2

You probably need to use https://www.elastic.co/guide/en/elasticsearch/reference/6.3/foreach-processor.html

3

just as @dadoonet recommended. Looking at foreach processor would help in this case.

here is the pipeline definition that I think you were intending.

PUT _ingest/pipeline/rename_model
{
  "processors": [
{
  "rename": {
    "field": "brand_id",
    "target_field": "brandId",
    "ignore_missing": true
  }
},
{
  "rename": {
    "field": "version.date_year",
    "target_field": "version.year",
    "ignore_missing": true
  }
},
{
  "foreach": {
    "field": "models",
    "processor": {
      "rename": {
        "field": "_ingest._value.model_id",
        "target_field": "_ingest._value.modelId",
        "ignore_missing": true
      }
    }
  }
}
  ]
}
4

Yes, its working. Thank you so much @dadoonet and @talevy. I have updated the pipeline and it is working like a charm.

Now, I have a follow up question. Can the foreach iterator have multiple processors in it? I mean say I want to iterate on an array, e.g., 'models' (here) and would like to perform 'rename', 'uppercase' etc.? Looks like I need 2 foreach for that, right?

5

Unfortunately, yes. At this moment, are limited to one processor per foreach. You would need two of them in this case

6

Okay @talevy, thanks a lot for your help!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.