Logstash mutate and copy entire _source object into an inner field


(Will Dombrowski) #1

Hello,

I'm trying to add a layer to my template definition to wrap the existing document as an inner field, lets call it 'summary'. I have to do this for several types, each with many fields. One of the bigger issues will be transforming all of the existing data to the new format. I am trying to use Logstash but am looking for a way to grab the whole _source instead of having to write each field by name.
Can someone suggest how to do this properly?

For example:

{
"firstName":"John",
"lastName":"Smith",
"middleName":"Adam"
}
filter {
  mutate {
    copy => {"firstName" => "summary.firstName"}
    copy => {"lastName" => "summary.lastName"}
    copy => {"middleName" => "summary.middleName"}
  }
}
{
    "summary": {
        "firstName":"John",
        "lastName":"Smith",
        "middleName":"Adam"
    }
}

(Magnus B├Ąck) #2

Use a ruby filter. Perhaps something like

event.set('summary', event.to_hash)

would work. If you don't want every single field to be copied into summary you can copy them one by one and use a conditional to select which ones to copy. For example,

summary = {}
event.to_hash.each { |k, v|
  if k.include? 'Name'
    summary[k] = v
  end
}
event.set('summary', summary)

only copies field whose name contains "Name".


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.