Ingest Processor for array object data using script

Elastic Stack Elasticsearch
1

Need assist to get the Ingest pipeline processors to work correctly for my sample data. Its an array of objects, trying to create individual objects from each array object using a value data + prefix names based on these values.

Input

{
   "color_bag": [{
       "val": "g1",
       "name": "color_type1",
       "attb": {}
    },
    {
       "val": "g2",
       "name": "color_type2",
       "attb": {}
    },
    {
       "val": "g3",
       "name": "color_type3",
       "attb": {}
    }]    
}

Output needed post processing

{
  "2025_red_g1": {
    "val": "g1",
    "name": "color_type1",
    "attb": {}
  },
  "2025_green_g2": {
    "val": "g2",
    "name": "color_type2",
    "attb": {}
  },
  "2025_blue_g3": {
    "val": "g3",
    "name": "color_type3",
    "attb": {}
  }
}

Logic in processor

  • For each object in array color_bag:
    • if color_bag.val = 'g1': then object_prefix = 'red_'
    • if color_bag.val = 'g2': then object_prefix = 'green_'
    • if color_bag.val = 'g3': then object_prefix = 'blue_'
  • Set: object_name = '2025_' + object_prefix + color_bag.val
  • Create new object as object_name, in parent root object with full data as in the its array object.

Sample Mapping

{
  "mappings": {
    "properties": {
      "2025_red_g1": {"type": "object", "properties" : {
          "val" : {"type": "keyword"},
          "name" : {"type": "keyword"},
          "attb": {"type": "object", "properties" : {"k": {"type": "keyword"}}}
       }},
      "2025_green_g2": {"type": "object", "properties" : {
          "val" : {"type": "keyword"},
          "name" : {"type": "keyword"},
          "attb": {"type": "object", "properties" : {"k": {"type": "keyword"}}}
       }},
      "2025_blue_g3": {"type": "object", "properties" : {
          "val" : {"type": "keyword"},
          "name" : {"type": "keyword"},
          "attb": {"type": "object", "properties" : {"k": {"type": "keyword"}}}
       }}
    }
  }
}

I have a half baked script which i know is wrong as it keeps erroring. Cannot get this coded correctly.

{
  "processors": [
    {
      "foreach": {
        "field": "color_bag",
        "processor": {
          "script": {
            "source": """
              def prefix = "2025_";

              if (ctx._ingest._value.val = 'g1') {
                  temp_name = 'red_';
              }
              if (ctx._ingest._value.val = 'g2') {
                  temp_name = 'green_';
              }
              if (ctx._ingest._value.val = 'g3') {
                  temp_name = 'blue_';
              }              
              
              def object_prefix = prefix + temp_name + ctx._ingest._value.val;
              ctx[object_prefix] = ctx._ingest._value;
            """
          }
        }
      }
    }
  ]
}
2

That type of dynamic naming of fields can result in excessive number of mapped fields, aka mapping explosion, and is generally a quite bad idea. I would recommend not syructuring your data that way as it is likely to cause problems unless the number of permutations is quite limited.

3

In total there are up to 8 permutations only as the max.