Split field and reverse join

illxjasper · December 6, 2016, 12:38pm

I am trying to parse appache accesslogs with logstash for a project.

To let elasticsearch search efficiënt I want to reverse the domainname. So for example:

www.example.com Becomes com.example.www

I tried to split the domainname and reverse that using the logstash mutate plugin. When parsed as is I get a field

domainname : "www.example.com"

Using settings below I get the following results:

Setting:

mutate{
    split => {"domainname" => "."}
    add_field => {"reversed_domainname" => ["%{[domainname][-1]}","%{[domainname][-2]}"]}
    join => {"reversed_domainname" => "."}
}

Result:

domainname = [ "www", "example", "com" ]

reversed_domainname =[ "com", "example" ]

Everything works as intended untill the join function as seen in the results i dont get com.example

If I remove reversed from the join (split the domainname on "." and join them with "." I get the same results.

www.example.com becomes www.example.com

How can I join the fields in reversed order as it clearly should work.

magnusbaeck · December 7, 2016, 6:48am

To let elasticsearch search efficiënt I want to reverse the domainname.

You should probably use the pathname analyzer on the ES side instead.

Contrary to popular belief, the options in a mutate filter are not executed in the order specificed; the order is rather fixed:

github.com

logstash-plugins/logstash-filter-mutate/blob/v3.1.3/lib/logstash/filters/mutate.rb#L202-L217


      
          def filter(event)
            rename(event) if @rename
            update(event) if @update
            replace(event) if @replace
            convert(event) if @convert
            gsub(event) if @gsub
            uppercase(event) if @uppercase
            lowercase(event) if @lowercase
            strip(event) if @strip
            remove(event) if @remove
            split(event) if @split
            join(event) if @join
            merge(event) if @merge
          
            filter_matched(event)
          end

Any add_field option executes that the end like for other filters. If you have mutations that depend on each other you need to use multiple mutate filters.

system · January 4, 2017, 6:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.