Dynamic fields name within grok match

[paska (Pavel Goncharov)]

Hello. I'm wanted to use dynamic fields name within grok match. But I couldn't find any information about it and about correct syntax.

First of all I'm getting field, which depends on field log.file.path

	grok {	
		match => { "log.file.path" => "(?<log_prefix>((?<=logs\/)(\w*)(?=\/)|(?<=logs\/)(\w*)(?=.*\.)))" }
	}

In the next step, I want to use value of log_prefix field as a part of new field:

grok {
			match => { "message" => "%{DATESTAMP:dev.%{log_prefix}.timestamp}] (?<dev.%{log_prefix}.loglevel>production\.[A-Z]{3,9}|local\.[A-Z]{3,9}): (?<dev.%{log_prefix}.message>(.|\r|\n)*)"}
	}

So, in this case, I've got error <Grok::PatternError: pattern %{log_prefix} not defined>. Of course I understand why. Is there any suggestion how I can do this correctly? Thanks.

[Cad]

Hi,

You should put the values in static fields and create the dynamic fields with the add_field option of the grok filter.
The code need to be like this i think:

grok {	
    match => { "log.file.path" => "(?<log_prefix>((?<=logs\/)(\w*)(?=\/)|(?<=logs\/)(\w*)(?=.*\.)))" }
}
grok {
    match => { "message" => "%{DATESTAMP:tmp1}] (?<tmp2>production\.[A-Z]{3,9}|local\.[A-Z]{3,9}): (?<tmp3>(.|\r|\n)*)"}
    #add_field executed only if the filter is successful
    add_field => {
        "dev.%{log_prefix}.timestamp" => "%{tmp1}"
        "dev.%{log_prefix}.loglevel" => "%{tmp2}"
        "dev.%{log_prefix}.message" => "%{tmp3}"
        }
	}

Cad.

[paska (Pavel Goncharov)]

Hi. Thanks for your solution. I've found another one for me. But it's similar. Instead of using "add_fields" I decided to use "mutate" filter to rename temporary fields.

grok {	
		match => { "log.file.path" => "(?<log_prefix>((?<=logs\/)(\w*)(?=\/)|(?<=logs\/)(\w*)(?=.*\.)))" }
}

grok {
		match => { "message" => "%{DATESTAMP:temp.timestamp}](\[(?<temp.thread>[0-9]*)\])? (?<temp.loglevel>production\.[A-Z]{3,9}|local\.[A-Z]{3,9}|Processing|Processed|Failed): (?<temp.message>(.|\r|\n)*[^\n]+)"}
		}

mutate {
			rename => { "temp.timestamp" => "%{server_type}.%{log_prefix}.timestamp" }
			rename => { "temp.loglevel"  => "%{server_type}.%{log_prefix}.loglevel"  }
			rename => { "temp.message"   => "%{server_type}.%{log_prefix}.message"   }
			rename => { "temp.thread"    => "%{server_type}.%{log_prefix}.thread"    }
}