The best practice to nest a hash?

lucianspec · June 30, 2017, 5:56am

my event data like this...

    "thread" => "Thread-74-__system-executor[-1 -1]",
   "message" => "2017-06-19 16:46:12.750 o.a.s.d.executor Thread-74-__system-executor[-1 -1] [INFO] Prepared bolt __system:(-1)",
      "type" => "o.a.s.d.executor",
   "content" => "Prepared bolt __system:(-1)",
 "timestamp" => "2017-06-19 16:46:12.750"

i want to put [timestamp, content, type, thread] replace the old message key & change the value type to a Hash

we found the best way is

mutate {
    add_field => {
        "[data][timestamp]" => "%{timestamp}"
        "[data][thread]" => "%{thread}"
    }
   remove_field =>  "message"
   # rename => {"data" => "message"}  #this will fail!!!
  }

mutate { rename => {"data" => "message"} }

this is so dirty....

data => {"thread" => "%{thread}", "timestamp" => "%{timestamp}"} will produce a array not a Hash...

so what's the best practice??? should we need to use ruby to convert a array to a Hash???

magnusbaeck · June 30, 2017, 8:03am

How about removing the existing message field and renaming the thread, timestamp, and content fields?

lucianspec · June 30, 2017, 8:15am

em....awesme idea...let me try
besides, Can you explain why rename a newly added field inside the same block will fail?? i am confused about it.

lucianspec · June 30, 2017, 8:43am

i am sorry it won't work..

 mutate { 
    remove_field => "message"
    rename => {
      "timestamp" => "[message][timestamp]"
    }
  }

"exception"=>"expecting List or Map, found class org.logstash.bivalues.StringBiValue",

but this will work

  mutate { 
    remove_field => "message"
  }
  mutate {
    rename => {
      "timestamp" => "[message][timestamp]"
      "thread" => "[message][thread]"
    }
  }

magnusbaeck · June 30, 2017, 9:20am

That's expected. Options in a mutate plugin aren't evaluated in the order specified but in this order:

github.com

logstash-plugins/logstash-filter-mutate/blob/v3.1.5/lib/logstash/filters/mutate.rb#L213-L229


def filter(event)
  rename(event) if @rename
  update(event) if @update
  replace(event) if @replace
  convert(event) if @convert
  gsub(event) if @gsub
  uppercase(event) if @uppercase
  lowercase(event) if @lowercase
  strip(event) if @strip
  remove(event) if @remove
  split(event) if @split
  join(event) if @join
  merge(event) if @merge
  copy(event) if @copy


  filter_matched(event)
end

system · July 28, 2017, 9:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.