Logstash Create new field from existing field and remove character from new field

Elastic Stack Logstash
1

I have a simple request but seem unable to be able execute it correctly.

I have a field named logname that has several name forms. For instance are logname-app, logname-os, logname-access, and logname. The logname is variable. The goal is to produce a variable logname in the path without the ending -app, -acess -os, etc and to use this in a logstash prefix. For instance, I would like to end with:
prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[sublog]}/application
prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[sublog]}/os
prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[sublog]}/access...etc.

I have the if statement working correctly based on the presence of the end of the field for instance.... if "-app" in [logname]....then moved data to and produce the one of the varients of the prefix above....

I most recently tried

filter {
if "-app" in [logname] [
mutate {
add field => {
"[sublog] => "%{[logname]}" }
remove_field => "-app"

the above does not work.

What I cannot get working is the filter or whatever to not only create the new sublog from the [logname], but remove the -app, etc from the newly formed sublog field so I can correctly insert the new sublog into the prefix without the -app at the end . Any help would be appreciated

2

Can you show using JSON a couple of examples of what your data looks like?

3

application1-app, application1-os, application1-access, application1-startup
application2-app, application2-os, application2-access, application2-startup
application3-app, application3-os, application3-access, application3-startup

I just want the application1, application2, application3 part of the field to show up in the prefix.

4

Assuming that that shows three separate events, and that they are the names of the fields then you would have to use ruby. I have not tested it but something like

ruby {
    code => '
        event.to_hash.each do |k, v|
            if k =~ /-app$/
                event.set("[@metadata][app]", k)
                break
        end
    '
}
grok { match => { "[@metadata][app]" => "^%{GREEDYDATA:[@metadata][prefix]}-app$" } }

[@metadata][prefix] will then contain "application1" or "application2", etc. You could then use

prefix => "%{+YYYY]/%{+MM}/%{+dd}/%{[@metadata][prefix]}/application

fields under [@metadata] are available in logstash, but outputs do not use them, so they are handy for storing data only needed inside the pipeline.

5

thanks...digesting this

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.