Remove part of field name from json inputted fields

Elastic Stack Logstash
1

I have an input into elasticsearch from a large amount of json formatted data. The input looks like this

  file{
    codec => json
    path => "/opt/apps/logs/health/*.log"
    type => "health"
  }
}

Very simple.

The data in each event looks like this (paraphrased)

First event

Item.1234.pineapple.red
Item.1234.pegasus.purple.dogsled```

Second Event
```Item.5555.description
Item.5555.pineapple.red
Item.5555.pegasus.purple.dogsled```

I would like to get rid of this randomly generate number in the 2nd array slot. 
In other words I want the results of each event to look like this

```Item.description
Item.pineapple.red
Item.pegasus.purple.dogsled```

I'm trying to use the ruby filter to do this, however it's not even close to working. Can anyone help me accomplish this task?
2

I have tried using the information posted here

And I just end up with _rubyparse errors in my tags as well as some additional "cannot convert string to integer " errors in my logstash.log

3

I'm trying to use the ruby filter to do this, however it's not even close to working. Can anyone help me accomplish this task?

Please show us what you have so far.

4

Ok, but fair warning I don't understand what I'm doing here, just trying to adapt your code from the previous post to this one.

filter {
  if [type] == "health"{
    ruby {
      code => "event['Item'].each {|key|
                        event['testing'] = key['description']
                     }"
    }
  }
}

This was me trying to keep it simple and just create a field named "testing" from the field Item.5656.description. Even that does not work.

5

This should work better:

filter {
  if [type] == "health"{
    ruby {
      code => "event['Item'].each_value {|value|
                        event['testing'] = value['description']
                     }"
    }
  }
}
1 Like
6

Thanks magnusbaeck, this worked but maybe I am approaching this wrong. Using this method I need to write a line for every field in the event

filter {
  if [type] == "health"{
    ruby {
      code => "event['Items'].each_value {|value|
                        event['description'] = value['description']
                        event['pineapple']['red'] = value['pineapple']['red']
                        event['pegasus']['purple']['dogsled'] = value['pegasus']['purple']['dogsled']
                     }"
    }
  }
}

Is this right? the whole reason I ended up here was because the mutate/rename plugin wouldn't support wildcards and I have 50 fields in each event that I need to remove the random number from. I can't seem to figure out how to automate the whole process.

I tried using slice, but no matter what I do with slice it tells me
Ruby exception occurred: undefined method `slice!' for #<Hash

7

Can't figure out why this doesn't work.

filter {
  if [type] == "health"{
    ruby {
      code => "event['Items'].each_value {|value|
                           event[value.delete_at(1)] = value.each
                     }"
    }
  }
}

Always results in
Ruby exception occurred: undefined method `delete_at' for #Hash:0x4453e3e5 {:level=>:error}

8

Inside the each_value loop, put another loop that iterates over the key/value pairs in value and copies them:

value.each_pair { |k, v|
  event[k] = v
}
1 Like
9

magnusblack
This is working properly, and I added some additional code (that I stole from another of your posts) that removes the old fields too. Here is the whole thing together.

filter {
  if [type] == "health"{
    ruby {
      code => "
        event['Item'].each_value {|value|
          value.each_pair { |key, val|
            event[key]=val
           }
         }
      "
    }
    ruby {
      code => "
        event.to_hash.keys.each { |k|
          if k.start_with?('Item')
            event.remove(k)
          end
        }
      "
    }
  }
}

magnusbaeck Not only to you have a kick ass name, but you have been a tremendous help to me. Thank you very much.