Logstash Create new field from existing field and remove character from new field

I have a simple request but seem unable to be able execute it correctly.

I have a field named logname that has several name forms. For instance are logname-app, logname-os, logname-access, and logname. The logname is variable. The goal is to produce a variable logname in the path without the ending -app, -acess -os, etc and to use this in a logstash prefix. For instance, I would like to end with:
prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[sublog]}/application
prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[sublog]}/os
prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[sublog]}/access...etc.

I have the if statement working correctly based on the presence of the end of the field for instance.... if "-app" in [logname]....then moved data to and produce the one of the varients of the prefix above....

I most recently tried

filter {
if "-app" in [logname] [
mutate {
add field => {
"[sublog] => "%{[logname]}" }
remove_field => "-app"

the above does not work.

What I cannot get working is the filter or whatever to not only create the new sublog from the [logname], but remove the -app, etc from the newly formed sublog field so I can correctly insert the new sublog into the prefix without the -app at the end . Any help would be appreciated

Can you show using JSON a couple of examples of what your data looks like?

application1-app, application1-os, application1-access, application1-startup
application2-app, application2-os, application2-access, application2-startup
application3-app, application3-os, application3-access, application3-startup

I just want the application1, application2, application3 part of the field to show up in the prefix.

Assuming that that shows three separate events, and that they are the names of the fields then you would have to use ruby. I have not tested it but something like

ruby {
    code => '
        event.to_hash.each do |k, v|
            if k =~ /-app$/
                event.set("[@metadata][app]", k)
                break
        end
    '
}
grok { match => { "[@metadata][app]" => "^%{GREEDYDATA:[@metadata][prefix]}-app$" } }

[@metadata][prefix] will then contain "application1" or "application2", etc. You could then use

prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[@metadata][prefix]}/application

fields under [@metadata] are available in logstash, but outputs do not use them, so they are handy for storing data only needed inside the pipeline.

thanks...digesting this

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.