Logstash Filter Theory

wendt · March 21, 2018, 12:59am

Do mutate functions build on each other? For example, if I were to rename a field and then wanted to send the text of that field to lower case, would I use the name of the original field, or the renamed version?

Ex. stacking functions
filter {
  mutate {
    rename => { "fruit" => "apple" }
    lowercase => [ "apple" ]
  }
}

Ex. non-stacking functions
filter {
  mutate {
    rename => { "fruit" => "apple" }
    lowercase => [ "fruit" ]
  }
}

Sorry for the formatting, still figuring out how to post here

leandrojmp · March 21, 2018, 6:57pm

The pipeline is read line by line, if you rename a field, everything else that you want to do with this field will need to use the new name.

This work without any problem.

filter {
  mutate {
    rename => { "fruit" => "apple" }
    lowercase => [ "apple" ]
  }
}

This does not work, since the field fruit does not exist anymore.

filter {
  mutate {
    rename => { "fruit" => "apple" }
    lowercase => [ "fruit" ]
  }
}

magnusbaeck · March 21, 2018, 8:17pm

The pipeline is read line by line

FIlters are processed in the order listed, yes, but within a filter the order shouldn't be relied upon. This is especially true for the mutate filter where the different operations run in a fixed order:

github.com

logstash-plugins/logstash-filter-mutate/blob/v3.3.1/lib/logstash/filters/mutate.rb#L247-L265


      
          def filter(event)
            coerce(event) if @coerce
            rename(event) if @rename
            update(event) if @update
            replace(event) if @replace
            convert(event) if @convert
            gsub(event) if @gsub
            uppercase(event) if @uppercase
            capitalize(event) if @capitalize
            lowercase(event) if @lowercase
            strip(event) if @strip
            remove(event) if @remove
            split(event) if @split
            join(event) if @join
            merge(event) if @merge
            copy(event) if @copy
          
            filter_matched(event)
          end

If you have operations that need to run in a particular order (like in the examples above) you must use separate mutate filters.

wendt · March 21, 2018, 8:25pm

Thank you for the explanation, that's extremely helpful!

Clarifying so I don't misrepresent this information in the future - the fixed order for mutate operations is as listed above (at least until the code is updated/changed), where coerce operations are run first, then rename operations, then update operations, and so on?

magnusbaeck · March 21, 2018, 8:26pm

Clarifying so I don't misrepresent this information in the future - the fixed order for mutate operations is as listed above (at least until the code is updated/changed), where coerce operations are run first, then rename operations, then update operations, and so on?

That's correct.

system · April 18, 2018, 8:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Adding a copy field and set it lowercase Logstash	3	1880	May 2, 2018
Mutate - Lowercase Logstash	2	2213	July 6, 2017
Logstash - Mutate lowercase doesn't work with filed name Logstash	2	636	June 23, 2020
Correct way to use the mutate rename filter Logstash	5	4999	June 30, 2021
Processing sequence for logstash - filter - mutate Logstash	1	572	May 2, 2019

Logstash Filter Theory

Related topics