Painless iterate over Nested Array

[Mandark1990 (Kshitij Thakkar)]

Hello Everyone,

Am trying to write an ingest pipeline which will help me normalize a nested array into individual fields for better visualization in Kibana. As my current event structure doesn't allow me to build the visualizations that I need.

For example below is the event that am ingesting am getting a field called tags which is a nested array as shown below:

"tags": [
          {
            "fields": [
              {
                "key": "host.name",
                "type": "string",
                "value": "localhost"
              },   
              {
                "key": "host.id",
                "type": "int64",
                "value": "21"
              },
              {
                "key": "status",
                "type": "string",
                "value": "OK"
              }
            ]
          }
        ]

What I need to do is convert all the key's within the tags.fields field into individual fields with the values from the value field and set the data type based on the type field.

I know one approach that I have used in the past and that is to match specific keys from the above structure and set them into new fields. However the problem is the keys within this array can increase or be renamed so I need to extract all keys into individual fields and not define them upfront.

 {
        "script": {
          "lang": "painless",
          "source": """
ctx.status= ctx.tags.stream().map(tag-> 
  tag.fields.stream()
    .filter(field -> field.key == "status")
    .map(field -> field.value)
    .collect(Collectors.toList())
  )
  .flatMap(l -> l.stream())
  .collect(Collectors.toList())"""
        }
      }

Has any one come across a similar situation and cracked this? Am open to any suggestions.