Flatten Array in Ingest Pipeline with Painless

[nicjp (Nic P)]

Hi all,

I'm working on getting some logs onboarded and I'm having an issue with a field that contains Key/Values inside an array. (For reference please see below)

{
  "redis_cache_read_bytes": 741,
  "format": "json",
  "params": [
    {
      "value": "backend",
      "key": "namespace_id"
    },
    {
      "value": "web-app",
      "key": "project_id"
    },
    {
      "value": "617",
      "key": "id"
    }
  ],
  ...
}

What I would like to be able to do is something like this:

{
  "redis_cache_read_bytes": 741,
  "format": "json",
  "item": {
    "backend": "namespace_id",
    "project_id": "web-app",
    "id": "617"
  }
}

I've considered a few ways of doing this (foreach processor into script processor, for loop via script processor etc. ) but I can't seem to make it do anything other than create a document with the last k/v value in the array (example below)

for (item in ctx.params) { ctx.item = item; }

{
  "item": {
    "value": "617",
    "key": "id"
  },
  "redis_cache_read_bytes": 741,
  "format": "json",
  "params": [
    {
      "value": "backend",
      "key": "namespace_id"
    },
    {
      "value": "web-app",
      "key": "project_id"
    },
    {
      "value": "617",
      "key": "id"
    }
  ],
...
}

[stu (Stuart Tettemer)]

Hi @nicjp, the code above is going through all params and assigning the current item, item, in the param list to ctx.item. This will overwrite whatever was in ctx.item, which is why you end up with the last k/v value.

Instead of that, you should create a new Map in ctx.item and update it with the values from the current item in the param list.

Try this:

ctx.item = new HashMap();
for (kv in ctx.params) {
  ctx.item[kv.get("key")] = kv.get("value");
}

[nicjp (Nic P)]

that is perfect - thank you @stu

I reached out to a former colleague (shout out to @Marius_Iversen) and he provided me with the following (very similar!) result.

 {
        "script": {
          "lang": "painless",
          "if": "ctx.params != null && ctx.params instanceof List",
          "source": """
          ctx.item = new HashMap();
          for (i in ctx.params) {
            ctx.item[i['key']] = i['value'];
          }
          """
        }
      }