Logstash mutate - rename field that occurs as an array

Elastic Stack Logstash
1

Hi, I'm trying to rename a field within an Elasticsearch type using the mutate filter. This particular field occurs within a zero-to-many array.

Here is a mapping that closely resembles mine (the names have been changed to protect the innocent):

"fieldRoot": {
  "properties": {
    "fieldCollection": {
      "properties": {
        "itemColor": {
          "index": "no",
          "type": "string"
        },
        "itemDescription": {
          "index": "no",
          "type": "string"
        }
      }
    }
  }
}

Here are some sample values:

"fieldRoot": {
   "fieldCollection": [
      {
         "itemColor": "blue",
         "itemDescription": "part 01"
      },
      {
         "itemColor": "white",
         "itemDescription": "part 02"
      },
      {
         "itemColor": "red",
         "itemDescription": "part 03"
      }
   ],
}

I want to rename itemDescription to itemLabel in the above mapping.
The mutate filter in my logstash script looks like this:

filter {
    mutate {
        rename => {"[fieldRoot][fieldCollection][0][itemDescription]" => "[fieldRoot][fieldCollection][0][itemLabel]"}
        rename => {"[fieldRoot][fieldCollection][1][itemDescription]" => "[fieldRoot][fieldCollection][1][itemLabel]"}
        rename => {"[fieldRoot][fieldCollection][2][itemDescription]" => "[fieldRoot][fieldCollection][2][itemLabel]"}
    }
}

This works fine if there are up to three members in the fieldCollection array. The problem is, there can be any number of occurrences of this array. Any further occurrences end up getting mapped to the old field name. I don't want to repeat dozens of lines in the mutate filter with the array ordinal as a hard coded sequence.

Is there a wildcard pattern I can use that substitutes for any number? Or is there a way to create a loop with a variable?

Thanks in advance!

2

Look into the ruby filter. Logstash's configuration language has no loop construct and no field name wildcarding.

3

Thanks Magnus. I suspected as much from what I read on similar posts. Thus begins my adventure into the world of Ruby. :head_bandage:

4

If there are any readers in suspense, this is what I ended up doing:

filter {
    if ([fieldRoot][fieldCollection]) {
        ruby {
            code => "event['fieldRoot']['fieldCollection'].each {|k|
                        k['itemLabel'] = k['itemDescription']
                        k['itemDescription'] = nil
                     }"
        }
    }
}

I couldn't prevent the old field name from being copied to the target index so I just gave it a null value. I was trying to wire up an event.remove(k) into the loop, but didn't have any luck. That would have been cleaner.

Does anyone know how to remove a field in a loop like this?

5

k is a Ruby hash so I'd expect k.delete('itemDescription') to work.

I suspect this will stop working in Logstash 5.0 which introduces a new event API (see https://github.com/elastic/logstash/issues/5141).

6

Perfect! Thanks!

No problem; we can definitely get some good mileage out if it until then.