Use mutate rename filter

[PitoneMaledetto (Giuseppe)]

Hi all,
I am having some trouble in renaming one of the output field.
Please consider (edited for easy reading):

[2018-09-14T13:04:43,936][DEBUG][logstash.pipeline        ] filter received
{"event"=>{"@timestamp"=>2018-09-14T12:04:40.194Z,
"system"=>{"load"=>{"norm"=>{"5"=>0.1363, "15"=>0.1175, "1"=>0.1763}, "5"=>1.09, "15"=>0.94, "1"=>1.41, "cores"=>8}},
"metricset"=>{"module"=>"system", "name"=>"load", "rtt"=>476},
"beat"=>{"name"=>"my-host-name ",
"hostname"=>"my-host-name ",
"version"=>"6.3.0"},
"@version"=>"1",
"host"=>{"name"=>"my-host-name "},
"tags"=>["metricbeat", "beats_input_raw_event"]}}

now I would like to rename (see attached) "norm" with "load_spread" and "1", "2" and "3" with "1_min_avg" and so on.

I have tried to get to the right field but nothing has changed, for example:

#--------------REPLACING NORM IN LOAD---------------
if "load" in [message] {
  mutate {
    rename => { "norm" => "load_avg" }
    #rename => { "norm_5" => "5_mins avg" }
    #rename => { "norm_15" => "15_mins_avg"}
  } 
}

Do I need to re-index? I don't get any errors in re-loading Logstash configuration so the lines are fine but I don't get any changes.
Any help appreciated.
Thanks!

[yaauie (Ry Biesemeyer)]

It looks like the address of the source norm field is nested inside a load field, which in turn is nested in a system field so you'll need to provide the entire address to the rename directive using the Field Reference Syntax:

if [system][load] {
  mutate {
    rename => {
      "[system][load][norm][1]"  => "[system][load_spread][1_min_avg]"
      "[system][load][norm][5]"  => "[system][load_spread][5_mins avg]"
      "[system][load][norm][15]" => "[system][load_spread][15_mins_avg]"
      "[system][load][1]"        => "[system][load][1_min_avg]"
      "[system][load][5]"        => "[system][load][5_mins avg]"
      "[system][load][15]"       => "[system][load][15_mins_avg]"
    }
    mutate {
      remove_field => "[system][load][norm]"
    }
  }
}

---

You will need to re-process your data, which may include reindexing. Many inputs hold onto metadata to prevent them from accidentally re-processing the same data, so you'll likely need to refer to your specific input plugin(s)' documentation to figure out how to force them to start from the beginning.

[PitoneMaledetto (Giuseppe)]

Thank you yaauie,
That worked in part.
It only process the data if I exclude:

    mutate {
      remove_field => "[system][load][norm]"
    }

otherwise:

####### STARTING FILTERS ###################################
#--------------REPLACING NORM IN LOAD---------------\nif [system][load]
{\n  mutate {\n    rename => {\n      
\"[system][load][norm][1]\"  => \"[system][load_spread][1_min_avg]\"\n 
\"[system][load][norm][5]\"  => \"[system][load_spread][5_mins avg]\"\n
\"[system][load][norm][15]\" => \"[system][load_spread][15_mins_avg]\"\n
\"[system][load][1]\"        => \"[system][load][1_min_avg]\"\n  
\"[system][load][5]\"        => \"[system][load][5_mins avg]\"\n 
\"[system][load][15]\"       => \"[system][load][15_mins_avg]\"\n    }
mutate ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'",
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'",
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'",
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/reload.rb:38:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

Thank you for your help.

[PitoneMaledetto (Giuseppe)]

Hi yaauie,
just to say that your solution is good enough for me and I only posted the above for completeness.
Thank you again for your help.
Best regards